{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dynamic recurrent neural networks for sequence classification with TensorFlow Eager\n",
    "----\n",
    "\n",
    "Hello everyone! In this tutorial, we are going to build a recurrent neural network for sentiment analysis on IMDB movie reviews. I've chosen this dataset as it is small in size and very easy to download by any of you so that there's no bottleneck in data acquisition :).\n",
    "\n",
    "The main aim of this tutorial is not focused on teaching you how to build a simple RNN, but rather on how to build a RNN that gives you more flexibility in the model development (e.g. use a new RNN cell that is not currently available in Keras, easier access to the unrolled outputs of the RNN, read data in batches from disk). My hope is to be able to give you a glimpse on how you can go on and build your own models, no matter how complicated they might get, in whatever area might be of interest to you.\n",
    "\n",
    "### Tutorial steps\n",
    "----\n",
    "![img](tutorials_graphics/04_flowchart.png)\n",
    "\n",
    "* *Download the **raw data** and transfer it to **TFRecords** (the default TensorFlow file format).*\n",
    "* *Prepare a **dataset Iterator** that reads data in batches, from disk, and automatically pads the input data of  variable length to the maximum size within a batch.*\n",
    "* *Build a **word-level RNN** model with both **LSTM** and **UGRNN cells**.*\n",
    "* *Compare the performance of both cells on the test dataset.*\n",
    "* **Save/restore trained model**\n",
    "* *Test the network on new reviews*\n",
    "* *Visualize RNN activations*\n",
    "\n",
    "\n",
    "If you would like to add anything to this tutorial please let me know. Also, I am happy to hear any suggestions you have for improvement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import useful libraries\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import functions to write and parse TFRecords\n",
    "from data_utils import imdb2tfrecords\n",
    "from data_utils import parse_imdb_sequence\n",
    "\n",
    "# Import TensorFlow and TensorFlow Eager\n",
    "import tensorflow as tf\n",
    "import tensorflow.contrib.eager as tfe\n",
    "\n",
    "# Import pandas for data processing and pickle for data reading\n",
    "import pandas as pd\n",
    "import pickle\n",
    "\n",
    "# Import library for plotting\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enable eager mode. Once activated it cannot be reversed! Run just once.\n",
    "tfe.enable_eager_execution(device_policy=tfe.DEVICE_PLACEMENT_SILENT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download data and transfer it to TFRecords\n",
    "----\n",
    "\n",
    "**Alternatively, if you clone this repository, you will automatically download the parsed data to TFRecords, so feel free to skip this step.** \n",
    "\n",
    "*Large Movie Review Dataset is 'a dataset for binary sentiment classification containing substantially more data than previous benchmark datasets'* (source [here](http://ai.stanford.edu/~amaas/data/sentiment/)). It is comprised of a train dataset with 25000 reviews (12500 positive and 125000 negative) and a test dataset with 25000 reviews as well (12500 positive and 125000 negative).\n",
    "\n",
    "Here is an example of a positive review:\n",
    "\n",
    "`\n",
    "Rented the movie as a joke. My friends and I had so much fun laughing at it that I went and found a used copy and bought it for myself. Now when all my friends are looking for a funny movie I give them Sasquatch Hunters. It needs to be said though there is a rule that was made that made the movie that much better. No talking is allowed while the movie is on unless the words are Sasquatch repeated in a chant. I loved the credit at the end of the movie as well. \"Thanks for the Jeep, Tom!\" Whoever Tom is I say thank you because without your Jeep the movie may not have been made. In short a great movie if you are looking for something to laugh at. If you want a good movie maybe look for something else but if you don't mind a laugh at the expense of a man in a monkey suit grab yourself a copy.\n",
    "`\n",
    "\n",
    "And here is an example of a negative review: \n",
    "\n",
    "`\n",
    "The Good: I liked this movie because it was the first horror movie I've seen in a long time that actually scared me. The acting wasn't too bad, and the \"Cupid\" killer was believable and disturbing. The Bad: The story line and plot of this movie is incredibly weak. There just wasn't much to it. The ways the killer killed his victims was very horrifying and disgusting. I do not recommend this movie to anyone who can not handle gore. Overall: A good scare, but a bad story.\n",
    "`\n",
    "\n",
    "To download this dataset, simply run in the terminal:\n",
    "> **chmod o+x datasets/get_imdb_dataset.sh**\n",
    "\n",
    "> **datasets/get_imdb_dataset.sh**\n",
    "\n",
    "Let's have a look at the distribution of the reviews' lengths (number of words/review):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEICAYAAABWJCMKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAF2FJREFUeJzt3X20XXV95/H3RxCsDzVBAosmkYea1RHXqsjcBcxyREdsCKlt6CxxUNcQKTNxZrBjl3ZZqLMGq9JBO1MHppU2lbTBJSCjdYguWsygoJ0KclFEMKUJyJhrUpL2BtShpUK/88f5XTlc7sO5yU3uw36/1jrr7P3dv7PP/uXc3M/dv/1wUlVIkrrnOXO9AZKkuWEASFJHGQCS1FEGgCR1lAEgSR1lAEhSRxkAUp8kDyd5wxy99/1JXjcX761uMgA0L8zFL94kf5zkQ4fyPadSVa+oqtvmejvUHQaANEuSHD7X2yDNhAGgeS/JG5Pck+TRJH+R5Gf7lj2c5NeS3JvksSSfSvK8vuXvTbI7ya4k/yZJJXlZkg3A24D3Jvlhks/1veUpk61v3Ha9Pcn/SfLRJKPA+1v9l5NsS7IvyS1Jjm/130/yX8et46Yk7+7ryxva9HOSXJLkwSR/m+TGJEe1ZZuTvKdNL299+g9t/mVJRtNzdJLPt3+30SRfSeL/ef2YPwya15KcCmwC3gG8BPgDYEuSI/uavRlYA5wI/Czw9vbaNcC7gTcALwNeO/aCqtoIfBL4SFW9sKp+Ybr1TeJ04CHgGODyJOcCvwH8S2AZ8BXg+tb2OuBfJUnbvqXAauCGCdb7H4Fz2zb/FLAP+L227HbgdW36te39x/p2JvCV6t3j5T3ASNuOY9t2ee8X/ZgBoPnu3wJ/UFV3VtVTVbUZeAI4o6/NVVW1q6pGgc8Bp7T6m4E/qqr7q+px4DcHfM/J1jeRXVX1P6rqyar6O3pB9V+qaltVPQn8Fr09iuPphUEBr2mvfRPw1araNcF63wG8r6pGquoJensXb2rDTLcDr2l/zZ8JfAR4dXvda9tygB8BxwHHV9WPqmosGCTAAND8dzzwnjaM8WiSR4GV9P4qHvPXfdOPAy9s0z8F7Oxb1j89lcnWN5Hx6zweuLJvW0eBAMvbL98bgLe0tm+ltxcykeOBz/atZxvwFHBsVT0I/JBeML0G+DywK8nP8MwA+G1gB/CFJA8luWSafqtjDADNdzuBy6tqSd/j+VV1/bSvhN3Air75leOWz8Zfw+PXsRN4x7jt/Ymq+ou2/Hp6f8kfT2/46DOTrHcncM649Tyvqr7Xlt9Obw/iiFa7HbgAWArcA1BVP6iq91TVScAvAO9OctYs9FmLhAGg+eS5SZ7X9zgc+EPg3yU5vR3YfEGSn0/yogHWdyNwYZKXJ3k+8J/HLX8EOGmW+/D7wKVJXgGQ5MVJzhtbWFXfAPYCHwduqapHp1jP5X0HkJclWde3/HbgncCX2/xtwK8Af15VT7XXvLEdFA7wfXp7EE/NTje1GBgAmk9uBv6u7/H+qhqmdxzgd+kdCN3B1Adlf6yq/hS4CvhSe91X26In2vM1wMltmOV/zUYHquqzwIeBG5J8H7gPOGdcs+vpHZi+bopVXQlsoTd88wPgDnp7DGNuB17E0wHw58Dz++YBVgH/m95w0VeBj3mdgfrFY0LqiiQvp/cL+ch2gFbqNPcAtKgl+aUkR7RTLj8MfM5f/lKPAaDF7h30xtwfpDf+/e/ndnOk+cMhIEnqKPcAJKmjpr15Vbu45FN9pZPonU53baufADwMvLmq9rVTzq4E1tK7iObtVfX1tq71wH9q6/lQu6pzUkcffXSdcMIJM+iOJOnuu+/+m6paNl27GQ0BJTkM+B6909EuBkar6op2heHSqvr1JGvpnY+8trW7sqpObzeyGgaG6F08czfwT6tq32TvNzQ0VMPDwwNvnyQJktxdVUPTtZvpENBZwINV9X+BdcDYX/Cb6d24ila/tnruAJYkOQ44G9haVaPtl/5WejfckiTNgZkGwPk8fWfDY6tqN0B7PqbVl/PM+6OMtNpkdUnSHBg4AJIcAfwi8D+nazpBraaoj3+fDUmGkwzv3bt30M2TJM3QTPYAzgG+XlWPtPlH2tAO7XlPq4/wzJturQB2TVF/hqraWFVDVTW0bNm0xzAkSftpJgHwFp4e/oHefUrWt+n1wE199QvajbvOAB5rQ0S3AKuTLO37IoxbDmjrJUn7baDvMG13Uvw5eldVjrkCuDHJRcB3gbE7Ht5M7wygHfROA70QoKpGk3wQuKu1+0D7wg1J0hyY11cCexqoJM3cwToNVJK0SBgAktRRAx0D6IzrJjpTFXjr/B0mk6T95R6AJHWUASBJHWUASFJHGQCS1FEGgCR1lAEgSR1lAEhSRxkAktRRBoAkdZQBIEkdZQBIUkcZAJLUUQaAJHWUASBJHWUASFJHGQCS1FEGgCR1lAEgSR1lAEhSRw0UAEmWJPl0kr9Msi3JP0tyVJKtSba356WtbZJclWRHknuTnNq3nvWt/fYk6w9WpyRJ0xt0D+BK4M+q6p8ArwS2AZcAt1bVKuDWNg9wDrCqPTYAVwMkOQq4DDgdOA24bCw0JEmH3rQBkOQngTOBawCq6h+q6lFgHbC5NdsMnNum1wHXVs8dwJIkxwFnA1urarSq9gFbgTWz2htJ0sAG2QM4CdgL/FGSbyT5eJIXAMdW1W6A9nxMa78c2Nn3+pFWm6z+DEk2JBlOMrx3794Zd0iSNJhBAuBw4FTg6qp6FfD/eHq4ZyKZoFZT1J9ZqNpYVUNVNbRs2bIBNk+StD8GCYARYKSq7mzzn6YXCI+0oR3a856+9iv7Xr8C2DVFXZI0B6YNgKr6a2Bnkp9ppbOAbwNbgLEzedYDN7XpLcAF7WygM4DH2hDRLcDqJEvbwd/VrSZJmgOHD9juV4BPJjkCeAi4kF543JjkIuC7wHmt7c3AWmAH8HhrS1WNJvkgcFdr94GqGp2VXkiSZmygAKiqe4ChCRadNUHbAi6eZD2bgE0z2UBJ0sHhlcCS1FEGgCR1lAEgSR1lAEhSRxkAktRRBoAkdZQBIEkdZQBIUkcZAJLUUQaAJHWUASBJHWUASFJHGQCS1FEGgCR1lAEgSR1lAEhSRxkAktRRBoAkdZQBIEkdZQBIUkcZAJLUUQaAJHXUQAGQ5OEk30pyT5LhVjsqydYk29vz0lZPkquS7Ehyb5JT+9azvrXfnmT9wemSJGkQM9kD+BdVdUpVDbX5S4Bbq2oVcGubBzgHWNUeG4CroRcYwGXA6cBpwGVjoSFJOvQOZAhoHbC5TW8Gzu2rX1s9dwBLkhwHnA1srarRqtoHbAXWHMD7S5IOwKABUMAXktydZEOrHVtVuwHa8zGtvhzY2ffakVabrP4MSTYkGU4yvHfv3sF7IkmakcMHbPfqqtqV5Bhga5K/nKJtJqjVFPVnFqo2AhsBhoaGnrVckjQ7BtoDqKpd7XkP8Fl6Y/iPtKEd2vOe1nwEWNn38hXArinqkqQ5MG0AJHlBkheNTQOrgfuALcDYmTzrgZva9BbggnY20BnAY22I6BZgdZKl7eDv6laTJM2BQYaAjgU+m2Ss/XVV9WdJ7gJuTHIR8F3gvNb+ZmAtsAN4HLgQoKpGk3wQuKu1+0BVjc5aTyRJMzJtAFTVQ8ArJ6j/LXDWBPUCLp5kXZuATTPfTEnSbPNKYEnqKANAkjrKAJCkjjIAJKmjDABJ6igDQJI6ygCQpI4yACSpowa9GVy3XTfRfeyAt3qvOkkLl3sAktRRBoAkdZQBIEkdZQBIUkcZAJLUUQaAJHWUASBJHWUASFJHGQCS1FEGgCR1lAEgSR1lAEhSRw0cAEkOS/KNJJ9v8ycmuTPJ9iSfSnJEqx/Z5ne05Sf0rePSVn8gydmz3RlJ0uBmsgfwLmBb3/yHgY9W1SpgH3BRq18E7KuqlwEfbe1IcjJwPvAKYA3wsSSHHdjmS5L210ABkGQF8PPAx9t8gNcDn25NNgPntul1bZ62/KzWfh1wQ1U9UVXfAXYAp81GJyRJMzfoHsB/B94L/GObfwnwaFU92eZHgOVtejmwE6Atf6y1/3F9gtf8WJINSYaTDO/du3cGXZEkzcS0AZDkjcCeqrq7vzxB05pm2VSvebpQtbGqhqpqaNmyZdNtniRpPw3yjWCvBn4xyVrgecBP0tsjWJLk8PZX/gpgV2s/AqwERpIcDrwYGO2rj+l/jSTpEJt2D6CqLq2qFVV1Ar2DuF+sqrcBXwLe1JqtB25q01vaPG35F6uqWv38dpbQicAq4Guz1hNJ0owcyHcC/zpwQ5IPAd8Armn1a4BPJNlB7y//8wGq6v4kNwLfBp4ELq6qpw7g/SVJB2BGAVBVtwG3temHmOAsnqr6e+C8SV5/OXD5TDdSkjT7vBJYkjrKAJCkjjIAJKmjDABJ6igDQJI6ygCQpI4yACSpowwASeooA0CSOsoAkKSOMgAkqaMMAEnqKANAkjrKAJCkjjIAJKmjDABJ6igDQJI6ygCQpI4yACSpowwASeooA0CSOsoAkKSOmjYAkjwvydeSfDPJ/Ul+s9VPTHJnku1JPpXkiFY/ss3vaMtP6FvXpa3+QJKzD1anJEnTG2QP4Ang9VX1SuAUYE2SM4APAx+tqlXAPuCi1v4iYF9VvQz4aGtHkpOB84FXAGuAjyU5bDY7I0ka3LQBUD0/bLPPbY8CXg98utU3A+e26XVtnrb8rCRp9Ruq6omq+g6wAzhtVnohSZqxgY4BJDksyT3AHmAr8CDwaFU92ZqMAMvb9HJgJ0Bb/hjwkv76BK/pf68NSYaTDO/du3fmPZIkDWSgAKiqp6rqFGAFvb/aXz5Rs/acSZZNVh//XhuraqiqhpYtWzbI5kmS9sOMzgKqqkeB24AzgCVJDm+LVgC72vQIsBKgLX8xMNpfn+A1kqRDbJCzgJYlWdKmfwJ4A7AN+BLwptZsPXBTm97S5mnLv1hV1ernt7OETgRWAV+brY5Ikmbm8OmbcBywuZ2x8xzgxqr6fJJvAzck+RDwDeCa1v4a4BNJdtD7y/98gKq6P8mNwLeBJ4GLq+qp2e2OJGlQ0wZAVd0LvGqC+kNMcBZPVf09cN4k67ocuHzmmylJmm1eCSxJHWUASFJHGQCS1FEGgCR1lAEgSR01yGmgmsx1E13cDLz1WRc4S9K84x6AJHWUASBJHWUASFJHGQCS1FHdPAg82cFbSeoQ9wAkqaMMAEnqKANAkjrKAJCkjjIAJKmjDABJ6igDQJI6ygCQpI4yACSpowwASeqoaQMgycokX0qyLcn9Sd7V6kcl2Zpke3te2upJclWSHUnuTXJq37rWt/bbk6w/eN2SJE1nkD2AJ4H3VNXLgTOAi5OcDFwC3FpVq4Bb2zzAOcCq9tgAXA29wAAuA04HTgMuGwsNSdKhN20AVNXuqvp6m/4BsA1YDqwDNrdmm4Fz2/Q64NrquQNYkuQ44Gxga1WNVtU+YCuwZlZ7I0ka2IyOASQ5AXgVcCdwbFXthl5IAMe0ZsuBnX0vG2m1yeqSpDkwcAAkeSHwGeBXq+r7UzWdoFZT1Me/z4Ykw0mG9+7dO+jmSZJmaKAASPJcer/8P1lVf9LKj7ShHdrznlYfAVb2vXwFsGuK+jNU1caqGqqqoWXLls2kL5KkGRjkLKAA1wDbqup3+hZtAcbO5FkP3NRXv6CdDXQG8FgbIroFWJ1kaTv4u7rVJElzYJBvBHs18K+BbyW5p9V+A7gCuDHJRcB3gfPaspuBtcAO4HHgQoCqGk3yQeCu1u4DVTU6K72QJM1Yqp41DD9vDA0N1fDw8OyveK6+EvKt8/ffWtLikeTuqhqarp1XAktSRxkAktRRBoAkdZQBIEkdZQBIUkcZAJLUUQaAJHWUASBJHWUASFJHGQCS1FEGgCR1lAEgSR1lAEhSRxkAktRRBoAkddQgXwij2TLZ9xD4PQGS5oB7AJLUUQaAJHWUASBJHWUASFJHGQCS1FEGgCR11LQBkGRTkj1J7uurHZVka5Lt7XlpqyfJVUl2JLk3yal9r1nf2m9Psv7gdEeSNKhB9gD+GFgzrnYJcGtVrQJubfMA5wCr2mMDcDX0AgO4DDgdOA24bCw0JElzY9oAqKovA6PjyuuAzW16M3BuX/3a6rkDWJLkOOBsYGtVjVbVPmArzw4VSdIhtL9XAh9bVbsBqmp3kmNafTmws6/dSKtNVn+WJBvo7T3w0pe+dD83b4HxCmFJc2C2DwJP9Juspqg/u1i1saqGqmpo2bJls7pxkqSn7W8APNKGdmjPe1p9BFjZ124FsGuKuiRpjuzvENAWYD1wRXu+qa/+ziQ30Dvg+1gbIroF+K2+A7+rgUv3f7MHNNnQiiRp+gBIcj3wOuDoJCP0zua5ArgxyUXAd4HzWvObgbXADuBx4EKAqhpN8kHgrtbuA1U1/sCyJOkQmjYAquotkyw6a4K2BVw8yXo2AZtmtHWSpIPGK4ElqaMMAEnqKL8RbD7z+gBJB5F7AJLUUQaAJHWUASBJHWUASFJHeRB4IfLgsKRZ4B6AJHWUASBJHWUASFJHeQxgMZnq7qceH5A0jnsAktRRBoAkdZRDQF3hqaOSxnEPQJI6yj2ArnPPQOos9wAkqaMMAEnqKIeANLGprimYiENG0oJjAGh2eCxBWnAMAB1cBoM0bx3yAEiyBrgSOAz4eFVdcai3QfPATIeYZsqAkaZ1SAMgyWHA7wE/B4wAdyXZUlXfPpTboQ5wz0Oa1qE+C+g0YEdVPVRV/wDcAKw7xNsgSeLQDwEtB3b2zY8Ap/c3SLIB2NBmf5jkgf14n6OBv9mvLVwY7N/+ettBHnoajJ/fwrVQ+nb8II0OdQBM9L/vGfvkVbUR2HhAb5IMV9XQgaxjPrN/C5v9W7gWW98O9RDQCLCyb34FsOsQb4MkiUMfAHcBq5KcmOQI4HxgyyHeBkkSh3gIqKqeTPJO4BZ6p4Fuqqr7D8JbHdAQ0gJg/xY2+7dwLaq+pcrT4iSpi7wZnCR1lAEgSR216AIgyZokDyTZkeSSud6e/ZHk4STfSnJPkuFWOyrJ1iTb2/PSVk+Sq1p/701y6txu/bMl2ZRkT5L7+moz7k+S9a399iTr56IvE5mkf+9P8r32Gd6TZG3fsktb/x5IcnZffV7+7CZZmeRLSbYluT/Ju1p9wX+GU/Rt0Xx+U6qqRfOgd2D5QeAk4Ajgm8DJc71d+9GPh4Gjx9U+AlzSpi8BPtym1wJ/Su8aizOAO+d6+yfoz5nAqcB9+9sf4Cjgofa8tE0vneu+TdG/9wO/NkHbk9vP5ZHAie3n9bD5/LMLHAec2qZfBPxV68eC/wyn6Nui+fymeiy2PYDFfKuJdcDmNr0ZOLevfm313AEsSXLcXGzgZKrqy8DouPJM+3M2sLWqRqtqH7AVWHPwt356k/RvMuuAG6rqiar6DrCD3s/tvP3ZrardVfX1Nv0DYBu9q/oX/Gc4Rd8ms+A+v6kstgCY6FYTU32Y81UBX0hyd7s1BsCxVbUbej+0wDGtvlD7PNP+LMR+vrMNgWwaGx5hgfcvyQnAq4A7WWSf4bi+wSL8/MZbbAEw7a0mFohXV9WpwDnAxUnOnKLtYunzmMn6s9D6eTXw08ApwG7gv7X6gu1fkhcCnwF+taq+P1XTCWrzuo8T9G3RfX4TWWwBsChuNVFVu9rzHuCz9HYvHxkb2mnPe1rzhdrnmfZnQfWzqh6pqqeq6h+BP6T3GcIC7V+S59L7BfnJqvqTVl4Un+FEfVtsn99kFlsALPhbTSR5QZIXjU0Dq4H76PVj7KyJ9cBNbXoLcEE78+IM4LGx3fJ5bqb9uQVYnWRp2x1f3Wrz0rjjML9E7zOEXv/OT3JkkhOBVcDXmMc/u0kCXANsq6rf6Vu04D/Dyfq2mD6/Kc31UejZftA7A+Gv6B2Rf99cb89+bP9J9M4g+CZw/1gfgJcAtwLb2/NRrR56X7LzIPAtYGiu+zBBn66ntxv9I3p/KV20P/0BfpneQbcdwIVz3a9p+veJtv330vtFcFxf+/e1/j0AnDPff3aBf05vOONe4J72WLsYPsMp+rZoPr+pHt4KQpI6arENAUmSBmQASFJHGQCS1FEGgCR1lAEgSR1lAEhSRxkAktRR/x/lXbVZLRdI3gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc037439ac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "length_reviews = pickle.load(open('datasets/aclImdb/length_reviews.pkl', 'rb'))\n",
    "pd.DataFrame(length_reviews, columns=['Length reviews']).hist(bins=50, color='orange');\n",
    "plt.grid(False);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that the majority of reviews have around 250 words. However, the distribution seems to have a really long tail due to some reviews that are quite long. As we will have to pad the variable length input sequence to the largest sequence in the batch, it will be quite ineficcient to keep these reviews. \n",
    "\n",
    "Thus, I have added an argument to the *imdb2tfrecords* function where you can specify the maximum number of words allowed in a review. This will simply take the last *max_words* in a review to make the training more efficient and avoid unrolling the neural network for many many timesteps, which can lead to memory problems. \n",
    "\n",
    "After downloading the dataset, simply run the *imdb2tfrecords* function. This function will parse each review into a list of word indices. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to process the raw data to TFRecords, indexing each word in the review to an integer index.\n",
    "#imdb2tfrecords(path_data='datasets/aclImdb/', min_word_frequency=5, max_words_review=700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of this processing, each tfrecord will be composed of:\n",
    "\n",
    "| TFrecord:        | Description           | \n",
    "|:----------------:|:---------------------:| \n",
    "| 'token_indexes'  | the sequence of word indexes present in the review|\n",
    "| 'target'      | 0 for negative sentiment or 1 for positive sentiment |\n",
    "| 'sequence_length' | the sequence length of the review.|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to test this RNN network with a new dataset, please have a look at the *imdb2tfrecords* or *parse_imdb_sequence* in the *data_utils.py* script in order to understand how you could parse your new data to TFRecords. I really recommend using this file format as it makes is very easy to work with very large datasets, without being constrained to your RAM capabilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a train and test dataset iterator\n",
    "----    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = tf.data.TFRecordDataset('datasets/aclImdb/train.tfrecords')\n",
    "train_dataset = train_dataset.map(parse_imdb_sequence).shuffle(buffer_size=10000)\n",
    "train_dataset = train_dataset.padded_batch(512, padded_shapes=([None],[],[]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_dataset = tf.data.TFRecordDataset('datasets/aclImdb/test.tfrecords')\n",
    "test_dataset = test_dataset.map(parse_imdb_sequence).shuffle(buffer_size=10000)\n",
    "test_dataset = test_dataset.padded_batch(512, padded_shapes=([None],[],[]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the word vocabulary\n",
    "word2idx = pickle.load(open('datasets/aclImdb/word2idx.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNN model for sequence classification, compatible with Eager API \n",
    "----\n",
    "In the cell below, you can find the class that I have created for the RNN model. The API is very similar with one I created in the previous tutorial, except that now we track the accuracy of the model instead of the loss.\n",
    "\n",
    "The idea of the network is very simple. We simply take each word in the review, select its corresponding word embedding (initialized randomly in the beginning), and pass it through the RNN cell. We then take the output of the RNN cell at the end of the sequence and pass it through a dense layer (with ReLU activation) to obtain the final predictions. \n",
    "\n",
    "Like usually, the network inherits from tf.keras.Model in order to keep track of all variables and save/restore them easily.\n",
    "\n",
    "![img](tutorials_graphics/rnn_imdb.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RNNModel(tf.keras.Model):\n",
    "    def __init__(self, embedding_size=100, cell_size=64, dense_size=128, \n",
    "                 num_classes=2, vocabulary_size=None, rnn_cell='lstm',\n",
    "                 device='cpu:0', checkpoint_directory=None):\n",
    "        ''' Define the parameterized layers used during forward-pass, the device\n",
    "            where you would like to run the computation on and the checkpoint\n",
    "            directory. Additionaly, you can also modify the default size of the \n",
    "            network.\n",
    "            \n",
    "            Args:\n",
    "                embedding_size: the size of the word embedding.\n",
    "                cell_size: RNN cell size.\n",
    "                dense_size: the size of the dense layer.\n",
    "                num_classes: the number of labels in the network.\n",
    "                vocabulary_size: the size of the word vocabulary.\n",
    "                rnn_cell: string, either 'lstm' or 'ugrnn'.\n",
    "                device: string, 'cpu:n' or 'gpu:n' (n can vary). Default, 'cpu:0'.\n",
    "                checkpoint_directory: the directory where you would like to save or \n",
    "                                      restore a model.\n",
    "        '''\n",
    "        super(RNNModel, self).__init__()\n",
    "        \n",
    "        # Weights initializer function\n",
    "        w_initializer = tf.contrib.layers.xavier_initializer()\n",
    "    \n",
    "        # Biases initializer function\n",
    "        b_initializer = tf.zeros_initializer()\n",
    "        \n",
    "        # Initialize weights for word embeddings \n",
    "        self.embeddings = tf.keras.layers.Embedding(vocabulary_size, embedding_size, \n",
    "                                                    embeddings_initializer=w_initializer)\n",
    "        \n",
    "        # Dense layer initialization\n",
    "        self.dense_layer = tf.keras.layers.Dense(dense_size, activation=tf.nn.relu, \n",
    "                                                 kernel_initializer=w_initializer, \n",
    "                                                 bias_initializer=b_initializer)\n",
    "        \n",
    "        # Predictions layer initialization\n",
    "        self.pred_layer = tf.keras.layers.Dense(num_classes, activation=None, \n",
    "                                                kernel_initializer=w_initializer, \n",
    "                                                bias_initializer=b_initializer)\n",
    "        \n",
    "        # Basic LSTM cell\n",
    "        if rnn_cell=='lstm':\n",
    "            self.rnn_cell = tf.nn.rnn_cell.BasicLSTMCell(cell_size)\n",
    "        # Else UGRNN cell\n",
    "        else:\n",
    "            self.rnn_cell = tf.contrib.rnn.UGRNNCell(cell_size)\n",
    "            \n",
    "        # Define the device \n",
    "        self.device = device\n",
    "        \n",
    "        # Define the checkpoint directory\n",
    "        self.checkpoint_directory = checkpoint_directory\n",
    "        \n",
    "    def predict(self, X, seq_length, is_training):\n",
    "        '''\n",
    "        Predicts the probability of each class, based on the input sample.\n",
    "\n",
    "        Args:\n",
    "            X: 2D tensor of shape (batch_size, time_steps).\n",
    "            seq_length: the length of each sequence in the batch.\n",
    "            is_training: Boolean. Either the network is predicting in\n",
    "                         training mode or not.\n",
    "        '''\n",
    "        \n",
    "        # Get the number of samples within a batch\n",
    "        num_samples = tf.shape(X)[0]\n",
    "\n",
    "        # Initialize LSTM cell state with zeros\n",
    "        state = self.rnn_cell.zero_state(num_samples, dtype=tf.float32)\n",
    "        \n",
    "        # Get the embedding of each word in the sequence\n",
    "        embedded_words = self.embeddings(X)\n",
    "        \n",
    "        # Unstack the embeddings\n",
    "        unstacked_embeddings = tf.unstack(embedded_words, axis=1)\n",
    "        \n",
    "        # Iterate through each timestep and append the predictions\n",
    "        outputs = []\n",
    "        for input_step in unstacked_embeddings:\n",
    "            output, state = self.rnn_cell(input_step, state)\n",
    "            outputs.append(output)\n",
    "            \n",
    "        # Stack outputs to (batch_size, time_steps, cell_size)\n",
    "        outputs = tf.stack(outputs, axis=1)\n",
    "        \n",
    "        # Extract the output of the last time step, of each sample\n",
    "        idxs_last_output = tf.stack([tf.range(num_samples), \n",
    "                                     tf.cast(seq_length-1, tf.int32)], axis=1)\n",
    "        final_output = tf.gather_nd(outputs, idxs_last_output)\n",
    "        \n",
    "        # Add dropout for regularization\n",
    "        dropped_output = tf.layers.dropout(final_output, rate=0.3, training=is_training)\n",
    "        \n",
    "        # Pass the last cell state through a dense layer (ReLU activation)\n",
    "        dense = self.dense_layer(dropped_output)\n",
    "        \n",
    "        # Compute the unnormalized log probabilities\n",
    "        logits = self.pred_layer(dense)\n",
    "        return logits\n",
    "    \n",
    "    def loss_fn(self, X, y, seq_length, is_training):\n",
    "        \"\"\" Defines the loss function used during \n",
    "            training.         \n",
    "        \"\"\"\n",
    "        preds = self.predict(X, seq_length, is_training)\n",
    "        loss = tf.losses.sparse_softmax_cross_entropy(labels=y, logits=preds)\n",
    "        return loss\n",
    "    \n",
    "    def grads_fn(self, X, y, seq_length, is_training):\n",
    "        \"\"\" Dynamically computes the gradients of the loss value\n",
    "            with respect to the parameters of the model, in each\n",
    "            forward pass.\n",
    "        \"\"\"\n",
    "        with tfe.GradientTape() as tape:\n",
    "            loss = self.loss_fn(X, y, seq_length, is_training)\n",
    "        return tape.gradient(loss, self.variables)\n",
    "    \n",
    "    def restore_model(self):\n",
    "        \"\"\" Function to restore trained model.\n",
    "        \"\"\"\n",
    "        with tf.device(self.device):\n",
    "            # Run the model once to initialize variables\n",
    "            dummy_input = tf.constant(tf.zeros((1,1)))\n",
    "            dummy_length = tf.constant(1, shape=(1,))\n",
    "            dummy_pred = self.predict(dummy_input, dummy_length, False)\n",
    "            # Restore the variables of the model\n",
    "            saver = tfe.Saver(self.variables)\n",
    "            saver.restore(tf.train.latest_checkpoint\n",
    "                          (self.checkpoint_directory))\n",
    "    \n",
    "    def save_model(self, global_step=0):\n",
    "        \"\"\" Function to save trained model.\n",
    "        \"\"\"\n",
    "        tfe.Saver(self.variables).save(self.checkpoint_directory, \n",
    "                                       global_step=global_step)   \n",
    "        \n",
    "    def fit(self, training_data, eval_data, optimizer, num_epochs=500, \n",
    "            early_stopping_rounds=10, verbose=10, train_from_scratch=False):\n",
    "        \"\"\" Function to train the model, using the selected optimizer and\n",
    "            for the desired number of epochs. You can either train from scratch\n",
    "            or load the latest model trained. Early stopping is used in order to\n",
    "            mitigate the risk of overfitting the network.\n",
    "            \n",
    "            Args:\n",
    "                training_data: the data you would like to train the model on.\n",
    "                                Must be in the tf.data.Dataset format.\n",
    "                eval_data: the data you would like to evaluate the model on.\n",
    "                            Must be in the tf.data.Dataset format.\n",
    "                optimizer: the optimizer used during training.\n",
    "                num_epochs: the maximum number of iterations you would like to \n",
    "                            train the model.\n",
    "                early_stopping_rounds: stop training if the accuracy on the eval \n",
    "                                       dataset does not increase after n epochs.\n",
    "                verbose: int. Specify how often to print the loss value of the network.\n",
    "                train_from_scratch: boolean. Whether to initialize variables of the\n",
    "                                    the last trained model or initialize them\n",
    "                                    randomly.\n",
    "        \"\"\" \n",
    "    \n",
    "        if train_from_scratch==False:\n",
    "            self.restore_model()\n",
    "        \n",
    "        # Initialize best_acc. This variable will store the highest accuracy\n",
    "        # on the eval dataset.\n",
    "        best_acc = 0\n",
    "        \n",
    "        # Initialize classes to update the mean accuracy of train and eval\n",
    "        train_acc = tfe.metrics.Accuracy('train_acc')\n",
    "        eval_acc = tfe.metrics.Accuracy('eval_acc')\n",
    "        \n",
    "        # Initialize dictionary to store the accuracy history\n",
    "        self.history = {}\n",
    "        self.history['train_acc'] = []\n",
    "        self.history['eval_acc'] = []\n",
    "        \n",
    "        # Begin training\n",
    "        with tf.device(self.device):\n",
    "            for i in range(num_epochs):\n",
    "                # Training with gradient descent\n",
    "                for X, y, seq_length in tfe.Iterator(training_data):\n",
    "                    grads = self.grads_fn(X, y, seq_length, True)\n",
    "                    optimizer.apply_gradients(zip(grads, self.variables))\n",
    "                    \n",
    "                # Check accuracy train dataset\n",
    "                for X, y, seq_length in tfe.Iterator(training_data):\n",
    "                    logits = self.predict(X, seq_length, False)\n",
    "                    preds = tf.argmax(logits, axis=1)\n",
    "                    train_acc(preds, y)\n",
    "                self.history['train_acc'].append(train_acc.result().numpy())\n",
    "                # Reset metrics\n",
    "                train_acc.init_variables()\n",
    "\n",
    "                # Check accuracy eval dataset\n",
    "                for X, y, seq_length in tfe.Iterator(eval_data):\n",
    "                    logits = self.predict(X, seq_length, False)\n",
    "                    preds = tf.argmax(logits, axis=1)\n",
    "                    eval_acc(preds, y)\n",
    "                self.history['eval_acc'].append(eval_acc.result().numpy())\n",
    "                # Reset metrics\n",
    "                eval_acc.init_variables()\n",
    "                \n",
    "                # Print train and eval accuracy\n",
    "                if (i==0) | ((i+1)%verbose==0):\n",
    "                    print('Train accuracy at epoch %d: ' %(i+1), self.history['train_acc'][-1])\n",
    "                    print('Eval accuracy at epoch %d: ' %(i+1), self.history['eval_acc'][-1])\n",
    "\n",
    "                # Check for early stopping\n",
    "                if self.history['eval_acc'][-1]>best_acc:\n",
    "                    best_acc = self.history['eval_acc'][-1]\n",
    "                    count = early_stopping_rounds\n",
    "                else:\n",
    "                    count -= 1\n",
    "                if count==0:\n",
    "                    break  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train model with gradient descent and early stopping\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model training with simple LSTM cells\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify the path where you want to save/restore the trained variables.\n",
    "checkpoint_directory = 'models_checkpoints/ImdbRNN/'\n",
    "\n",
    "# Use the GPU if available.\n",
    "device = 'gpu:0' if tfe.num_gpus()>0 else 'cpu:0'\n",
    "\n",
    "# Define optimizer.\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=1e-4)\n",
    "\n",
    "# Instantiate model. This doesn't initialize the variables yet.\n",
    "lstm_model = RNNModel(vocabulary_size=len(word2idx), device=device, \n",
    "                      checkpoint_directory=checkpoint_directory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy at epoch 1:  0.72308\n",
      "Eval accuracy at epoch 1:  0.68372\n",
      "Train accuracy at epoch 2:  0.77708\n",
      "Eval accuracy at epoch 2:  0.75472\n",
      "Train accuracy at epoch 3:  0.875\n",
      "Eval accuracy at epoch 3:  0.82036\n",
      "Train accuracy at epoch 4:  0.91728\n",
      "Eval accuracy at epoch 4:  0.8542\n",
      "Train accuracy at epoch 5:  0.94728\n",
      "Eval accuracy at epoch 5:  0.87464\n",
      "Train accuracy at epoch 6:  0.96312\n",
      "Eval accuracy at epoch 6:  0.88228\n",
      "Train accuracy at epoch 7:  0.97476\n",
      "Eval accuracy at epoch 7:  0.88624\n",
      "Train accuracy at epoch 8:  0.9828\n",
      "Eval accuracy at epoch 8:  0.88344\n",
      "Train accuracy at epoch 9:  0.98692\n",
      "Eval accuracy at epoch 9:  0.87036\n",
      "Train accuracy at epoch 10:  0.99052\n",
      "Eval accuracy at epoch 10:  0.86724\n",
      "Train accuracy at epoch 11:  0.9944\n",
      "Eval accuracy at epoch 11:  0.87088\n",
      "Train accuracy at epoch 12:  0.99568\n",
      "Eval accuracy at epoch 12:  0.86068\n"
     ]
    }
   ],
   "source": [
    "# Train model\n",
    "lstm_model.fit(train_dataset, test_dataset, optimizer, num_epochs=500, \n",
    "                early_stopping_rounds=5, verbose=1, train_from_scratch=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save model\n",
    "lstm_model.save_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model training with UGRNN cells \n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define optimizer.\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=1e-4)\n",
    "\n",
    "# Instantiate model. This doesn't initialize the variables yet.\n",
    "ugrnn_model = RNNModel(vocabulary_size=len(word2idx), rnn_cell='ugrnn', \n",
    "                       device=device, checkpoint_directory=checkpoint_directory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy at epoch 1:  0.71092\n",
      "Eval accuracy at epoch 1:  0.67688\n",
      "Train accuracy at epoch 2:  0.82512\n",
      "Eval accuracy at epoch 2:  0.7982\n",
      "Train accuracy at epoch 3:  0.88792\n",
      "Eval accuracy at epoch 3:  0.84116\n",
      "Train accuracy at epoch 4:  0.92156\n",
      "Eval accuracy at epoch 4:  0.85076\n",
      "Train accuracy at epoch 5:  0.94592\n",
      "Eval accuracy at epoch 5:  0.86476\n",
      "Train accuracy at epoch 6:  0.95984\n",
      "Eval accuracy at epoch 6:  0.87104\n",
      "Train accuracy at epoch 7:  0.9708\n",
      "Eval accuracy at epoch 7:  0.87188\n",
      "Train accuracy at epoch 8:  0.9786\n",
      "Eval accuracy at epoch 8:  0.8748\n",
      "Train accuracy at epoch 9:  0.98412\n",
      "Eval accuracy at epoch 9:  0.86452\n",
      "Train accuracy at epoch 10:  0.9882\n",
      "Eval accuracy at epoch 10:  0.86172\n",
      "Train accuracy at epoch 11:  0.9938\n",
      "Eval accuracy at epoch 11:  0.86808\n",
      "Train accuracy at epoch 12:  0.9956\n",
      "Eval accuracy at epoch 12:  0.8596\n",
      "Train accuracy at epoch 13:  0.997\n",
      "Eval accuracy at epoch 13:  0.86368\n"
     ]
    }
   ],
   "source": [
    "# Train model\n",
    "ugrnn_model.fit(train_dataset, test_dataset, optimizer, num_epochs=500, \n",
    "                early_stopping_rounds=5, verbose=1, train_from_scratch=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance comparison\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl0AAAD8CAYAAABNXRFJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xd8VFX+//HXSe+9k4SE3kJICL2JSl3FsroiVtRFXdT9uusWld9adl11V93V1RVdxS42LKCIBSkB6b2TQAKkQXpvM3N+f9whjQAJJJlJ8nk+HvOYdu/MJxAO73vuuecorTVCCCGEEKJ9Odi6ACGEEEKI7kBClxBCCCFEB5DQJYQQQgjRASR0CSGEEEJ0AAldQgghhBAdQEKXEEIIIUQHkNAlhBBCCNEBJHQJIYQQQnQACV1CCCGEEB3AydYFNBUUFKRjYmJsXYYQogNt27YtT2sdbOs62oK0YUJ0L61pv+wudMXExLB161ZblyGE6EBKqWO2rqGtSBsmRPfSmvZLTi8KIYQQQnQACV1CCCGEEB1AQpcQQgghRAc4b+hSSi1SSp1SSu09y/tKKfWSUipVKbVbKZXY4L3blFIp1tttbVm4EEIIIURn0pKerreB6ed4fwbQ13qbB7wKoJQKAB4DRgEjgceUUv4XU6wQQgghRGd13tCltV4LFJxjk6uAd7VhI+CnlAoHpgE/aK0LtNaFwA+cO7wJIYQQQnRZbTGmqwdwosHzDOtrZ3v9DEqpeUqprUqprbm5uW1QkhBCdBxpw4QQLdEW83SpZl7T53j9zBe1fh14HSApKanZbYQQwl5JGyaEfdNaU1lrpqLGTEW1mYpaE+XVZiprzJTXmBrfV5uJDfZkVnxEm9fRFqErA4hq8DwSyLK+fkmT11e3wfcJIdqJyWyhymShssZMVa1xq6w1U1VrobLWaKCqTcZ9w9frtq0x1+1/36V9GBblZ+sfSQjRSVksmvIaE6VVp2+1lFY3eNzgvqzKRHmNyQhVNWbKq01U1pqtwcpERa0Z3YrDoZlxYXYbupYC9ymlPsIYNF+stc5WSn0H/L3B4PmpwMNt8H1CiFawWDR5ZdVkFVeRXVRJZlEl2cVVZBdXklVURU5xFeU1JqpqzdSaL6yTxs3ZATdnR9ytN1dnRypqTG38kwghOrtqk5nUU2UczC7lSG4ZRZW1lDUKUQ2CVI3pvEHJ0UHh7eaEl6txc3dxxNPFiQBPFzxdHHF3ccLTxREPF0c8XJ2Me5fT940fe1r393B2xMmxfWbUOm/oUkotxuixClJKZWBckegMoLVeCCwHZgKpQAUw1/pegVLqr8AW60c9qbU+14B8IUQraa0prqwlq8gaooqryCqqJLvIeJxdXElOcdUZYcrd2ZFwPzcifN0Z3zeorrFyd3bEzdnBeu9YH6RcHBsFq4b3rk4OODg0N5pACNFdaa05WVLNgZwSDmSXcDC7lIM5JRzJLcdsMdojZ0eFr7sz3m7OeLk64e3mREyQB95uzni7OeHt6lT/uO7eqdFzd2dHlOo87c95Q5fW+sbzvK+B+Wd5bxGw6MJKE6J701pTVFFLTkkVOSVVnCyuquuhyj4droqrqKgxN9rPyUER5msEqsRofyL83InwdSPc151wPzd6+Lnj6+7cqRoqIYT9qqwxc/ikEaoOWMPVwZxSiipq67bp4efOwHBvpg4KY0C4NwPCfIgJ9Gi3HiV7ZXcLXgvRHdSYLJwqreJkSRU5xdVGqCoxQtXJ4qq659Umyxn7Bnu7EuHrRr9Qbyb1CyHCz40IP3fCfY37IC9XHKXnSQjRxiwWTWZRJQdzSjmYXcKBHKMHKy2/vO40oIeLI/3DvJkxJJyB1nDVP8wbX3dn2xZvJyR0CdHGqmrNZDUYO9UwROWUGGOo8spqztjPxcmBMB83wnzciI/yI9zXjVDr8zBfV0J93AjxdsPFqXsdGQohOk5xZS0nCiqMW2EFxwsqOF5QSUZBBRmFldSYjQNBpaBngAcDwnyYNSyCAWE+DAz3JsrfQ4YbnIOELiFaqaSqlszCSuNmDVeZhZVkWO/zyqrP2Mffw9kIUL5uDInwrXsc1uDez0NO+Qkh2leNyUJmUSUnCoxAdaKwov5xQSXFlbWNtvfzcCbK34OB4T5MGRxKTKAnA8K86RfqjaerRIjWkj8xIRrQWpNfXmOEqMJKMosq6sJVhvW+tKrxVXkuTg708HOnh587lw0IoYe/8biHv3tdb5Wbs6ONfiIhRHejtSa7uIpDOaUcyCkhLbec49aequziSiwNrqtxcXQgMsCdKH8PEqL8iQpwJzrAgyjrzcdNTgu2JQldotuqqjWzO6OYLekFbDtWSHp+OVlFlVTVNh5H5e3qVBekRsYG1AWq0/dBnq7SnS6EsInSqlrrIPZSDmaXcijHGMhe0uDgMMTblZ6BHoyKDagLU9HWW4i3tF8dSUKX6DaKK2vZdqyALemFbEkrYHdGcd34hD4hXvQP9ebS/iFE+rvTw9+jLlTJAFAhhK2ZzBbS8so5mFMfrA7mlJJRWFm3jberE/3DvLkyPoIB4T51pwGlDbMfErpEl5VdXFkXsLakF3DoZClaG1MqxEX6cvu4GEbEBJDU0x9/TxdblyuEEADklVWzL6uEQ9ZgdTC7lNTcMmqsVzM7Oih6BXmSEO3PjSOjGRDmTf8wb3r4ucu4UDsnoUt0CVprjuSWsTmtkK3pBWxOL6g7AvR0cSSxpz8z48JJivEnIcofdxcZYyWEsA95ZdVsOlrAxqP5bDyaT8qpsrr3Qn1c6R/mw4S+QfS3hqs+IV64Okkb1hlJ6BKdUq3Zwt7MYramF7I5vYCt6QUUWifiC/JyYURMAHPHxTIyJoCB4d7dbgI+IYT9yi+rZlNafcg6fNIIWR4ujoyICeDaxEjio3wZGOYjvfBdjIQu0WkUltewYl8Oy/dkszW9kMpaYyb2noEeXDYwlJExASTF+BMb5Cld7EIIu5FfVs3mtAI2NBOykmICuCYhktG9AhjSwxdnOUDs0iR0CbtWWlXLD/tPsmxXFskpeZgsmphAD24YEcWImABGxPgT4uNm6zKFEKJOQXkNm6wBa+NRYzwp1IesqxN6MLpXIHESsrodCV3C7lTWmFl50Ahaqw7lUmOy0MPPnTvHx3JlfASDI3ykJ0sIYTeqas2sOZzLhiNG0DqYY4Qsd2dHkmL8mTUsgtG9AhkaKSGru5PQJexCtcnM2sN5LNuVxY8HTlJRYybY25U5I6O5Mj6chCh/mUtGCGE3TGYLPx/J56udWXy3L4eyalNdyLoyXkKWaJ6ELmEztdZGa9kuo9EqrTLh7+HM1Qk9uGJoOKNiA2XhZiGE3dBas/NEEV/tzOLr3VnkldXg7ebEzLgwZsX3YGRsgKyNKs5JQpfoUGaLZnNaAct2Z7Fibw4F5TV4uzoxdXAYV8aHM65PkBwZCiHsSuqpUr7amcVXO7M4XlCBi5MDlw8MYVZ8Dy7pHyzLfIkWk9Al2p3Wmu3Hi/h6dxbf7M7mVGk17s6OXD4olCuHhjOxnzRaQgj7kl1cybJdWXy5I4v92SU4KBjXJ4j7L+3DtCFhsiahuCASukS7OVVSxTsb0vlyRxaZRZW4ODkwuX8wV8ZHcOmAEDxc5NdPCGE/iipqWL4nh692ZrI5vQCtIT7Kj8euHMQvhoYT4i1XSouLI//riTaXllfO62uPsGRbJiaLhYn9gvndlH5MGRwqR4dCCLtSWWPmhwMnWbozkzWHc6k1a3oFe/Lg5f2YFR9BTJCnrUsUXYiELtFmdmcUsXDNEb7dm4OzowPXJ0Uyb2IvegZKoyWEsC/bjhXy/sZjfLcvh4oaM2E+bswdF8ssmZZGtCMJXeKiaK1Zn5rPq2tSWZ+aj7ebE/dO6s3ccbEEe7vaujwhhKhjMlv4fv9J/pd8lB3Hi/B2c+KqYRF1Vx7K1dKivUnoEhfEbNF8uzebhWuOsDezhBBvVx6ZOYAbR0bjLacQhRB2pKzaxCdbTrBofRoZhZX0DPTgiVmDuW54JJ6u8t+g6Djy2yZaparWzJLtGfxv7VHS8yvoFeTJM9fGcU1iD1n1XghhV7KKKnn753QWbzpOabWJETH+LPjFIKYMCpVeLWETLQpdSqnpwIuAI/CG1vqZJu/3BBYBwUABcLPWOsP6nhnYY930uNZ6VhvVLjpQSVUt7288xqJ16eSVVRMf6cvCmxOZMihMGi8hhF3Zk1HMG+uO8s3ubDQwY0gYd03oxbAoP1uXJrq584YupZQj8AowBcgAtiillmqt9zfY7DngXa31O0qpS4GngVus71VqrYe1cd2ig5wqqeLN9Wl8uNE4UpzQN4h7Jw1jTO9AGWgqhLAbFotm5cFTvJF8lE1pBXi5OnH72BhuHxdDpL+HrcsTAmhZT9dIIFVrfRRAKfURcBXQMHQNAh60Pl4FfNmWRYqO13Tah5lx4dwzqTdDevjaujQhhKhTWWPms+0ZLFqXRlpeOT383Fnwi4H8akSUTFEj7E5LQlcP4ESD5xnAqCbb7AJ+iXEK8hrAWykVqLXOB9yUUlsBE/CM1vqMQKaUmgfMA4iOjm71DyHajkz7IETrSRvW8U6VVvHuz8d4f9MxiipqGRrpy0s3JjBzSBhOspSYsFMtCV3NnUPSTZ4/BLyslLodWAtkYoQsgGitdZZSqhfwk1Jqj9b6SKMP0/p14HWApKSkpp8tOkBxZS2/+3gnKw+ekmkfhGglacM6zsGcEt5ITmPpzixqLRamDAzlrgm9GBHjL0MehN1rSejKAKIaPI8EshpuoLXOAq4FUEp5Ab/UWhc3eA+t9VGl1GogAWgUuoRtZRVVcvtbm0nLK+cP0/pz65ieMu2DEMKuVNWa+ctXe/lkawbuzo7MHhnF3HGxxMqM8aITaUno2gL0VUrFYvRgzQbmNNxAKRUEFGitLcDDGFcyopTyByq01tXWbcYB/2jD+sVFOpBdwu1vbaai2sw7c0cytk+QrUsSQohGThRUcPd729ifXcLdk3px76Te+Hm42LosIVrtvKFLa21SSt0HfIcxZcQirfU+pdSTwFat9VLgEuBppZTGOL0437r7QOA1pZQFcMAY07X/jC8RNrE+NY+739uGl6sTn9wzhoHhPrYuSQghGll16BT/99FOtNa8eVsSlw0MtXVJQlywFs3TpbVeDixv8tpfGjz+DPismf1+BuIuskbRDr7YkcEfP9tNryAv3r5jBOG+7rYuSQgh6lgsmv/8lMq/Vx6mf6g3r90yXC7oEZ2ezEjfzWiteXXNEf6x4hCjewXw2i1J+LrL+C0hhP0orqjlwU928tPBU1yT0IO/XxOHu4useCE6Pwld3YjZonls6V7e33icWfER/PP6obJ0jxDCruzPKuGe97eRVVTJk1cN5pbRPeWqRNFlSOjqJiprzNy/eAc/HjjJPZN688dp/XGQ5XuEEHbkix0ZPPz5Hnzdnfn47tEM7xlg65KEaFMSurqB/LJq7nxnK7syinjyqsHcOibG1iUJIUSdGpOFv32zn3c3HGNkbAAvz0kgxNvN1mUJ0eYkdHVxx/LLuW3RZrKLq3j1puFMHxJm65KEEKJOTnEVv/lgG9uPF3HX+Fj+NGMAzjKjvOiiJHR1YTtPFHHn21uwaM2Hvx7N8J7+ti5JCCHqbDyaz30fbqeixszLcxK4YmiErUsSol1J6OqiVh44yfwPtxPs7co7c0fSK9jL1iUJIQRgXEX95ro0nv72ID0DPFj869H0DfW2dVlCtDsJXV3QB5uO8f++3MvgCF8W3T5C1k8UQtiN8moTf1yym292ZzNtcCjPXR8vy46JbkNCVxeiteb57w/z8qpUJvcP5uU5iXi6yl+xEMI+HMkt4+73tnE0t4w/TR/APZN6yXQQoluR/5G7iBqThT9/vpvPt2dy48go/nrVEJxkMKoQwk6s2JvNQ5/uxsXJgffuHMU4WedVdEMSurqA0qpafvPBdpJT8vjdlH7cf2kfOXoUQtgFk9nCc98fZuGaI8RH+vLfm4fTw0+WHRPdk4SuTu5kSRW3v7WFlJOl/PO6oVyfFGXrkoQQAjB64O95fxs/HTzFnFHRPHblIFkFQ3RrEro6sZSTpdz+1haKKmp48/YRTOoXbOuShBACMJYd+/2nu/jp4CmZlFkIKwldndSmo/n8+t2tuDo78vHdYxjSw9fWJQkhBGBc1PPY0r0s25XFn2cMkMAlhJWErk7oYE4Jc9/eQrivG2/PHUlUgIetSxJCiDov/HCY9zce5+6JvbhnUm9blyOE3ZDQ1ckUVdQw791teLo68eGvRxPqI+uTCSHsx5vr0vjPT6nckBTFn2cMsHU5QtgVmVOgEzFbNPcv3kF2cSULbx4ugUsIYVeWbMvgr1/vZ/rgMJ66ZohcRS1EE9LT1Yn847uDJKfk8cy1cbKOohDCrvyw/yR/XLKbcX0CefHGYTJPoBDNkH8VncTSXVm8tuYoN42KZvbIaFuXI4QQdTYezWf+h9sZEuHDa7ckybQQQpyFhK5OYH9WCX/8bBdJPf157MrBti5HCCHq7M0s5q53thId4MFbc0fiJUuPCXFWErrsXGF5DfPe24qfuwv/vTkRFyf5KxNC2IejuWXctmgzvu7OvHfnSAI8XWxdkhB2rUX/gyulpiulDimlUpVSf27m/Z5KqZVKqd1KqdVKqcgG792mlEqx3m5ry+K7OpPZwn2Lt3OqpJqFtwwnxFsGzgsh7EN2cSW3vLkZgPfuHEm4ryztI8T5nDd0KaUcgVeAGcAg4Eal1KAmmz0HvKu1Hgo8CTxt3TcAeAwYBYwEHlNKyQjwFnrm24OsT83nb9cMYViUn63LEUIIAArKa7jlzc0UV9byzh0j6RXsZeuShOgUWtLTNRJI1Vof1VrXAB8BVzXZZhCw0vp4VYP3pwE/aK0LtNaFwA/A9Isvu+v7ckcmb6xL47YxPfmVrKcohLATZdUm5r61mRMFFbxxW5KshiFEK7QkdPUATjR4nmF9raFdwC+tj68BvJVSgS3cVzSxN7OYPy3ZzcjYABZc0bRTUQghbKPaZObu97ayN6uEV+YkMrpXoK1LEqJTaUnoam52O93k+UPAJKXUDmASkAmYWrgvSql5SqmtSqmtubm5LSip68ovq+bu97YR4OnCf29KxFnmuhHC7nWHNsxs0fx28U7Wp+bzj18O5fJBobYuSYhOpyX/o2cADc9vRQJZDTfQWmdpra/VWicAj1pfK27JvtZtX9daJ2mtk4KDg1v5I3QdtWYL8z/cTm5ZNa/dMpwgL1dblySEaIGu3oZprXnk8z2s2JfD/7tiEL8cHnn+nYQQZ2hJ6NoC9FVKxSqlXIDZwNKGGyilgpRSpz/rYWCR9fF3wFSllL91AP1U62uiGX9ffoCNRwt4+po4hkbKwHkhhH14ZsVBPt56gvsv7cOd42NtXY4QndZ5Q5fW2gTchxGWDgCfaK33KaWeVErNsm52CXBIKXUYCAWesu5bAPwVI7htAZ60viaaWLItg7fWpzN3XIwcRQoh7MbCNUd4bc1Rbhndk99N6WfrcoTo1Fo0dbDWejmwvMlrf2nw+DPgs7Psu4j6ni/RjN0ZRTz8xR7G9ArkkZkDbV2OEEIA8NHm4zzz7UGujI/giVmDZQFrIS6SjNK2sdxSY+B8sJcrL89JkIHzQgi78O2ebB75Yg+T+gXz/PXxODhI4BLiYskiWTZUa7Yw/4PtFFbU8Nk9YwmUgfNCCDuwLiWP3360k4Rof16V5ceEaDPyL8mG/vr1fjanF/DsL4fKBIMtZbFAWS5UlYA+Y/YRIcRFKiyv4d4PttEr2JNFt43Aw0WOzYVoK/KvyUY+2XKCdzcc49cTYrlqmMwXW6e6DEoyofgEFGdAcab13vq8JBPMNca2ji7gEQSegdb7oLM8DzZec/MDGZMixDktWp9GaZWJF2cn4OvhbOtyhOhSJHTZwI7jhSz4ci/j+wTxp+kDbF1Ox7GYoTSncYg6HaROP68sbLyPcgDvcPCNhB6JMGgW+PQAUzVU5EF5vvU+DwrTjOc1pc1/v4MTeAQ2H8y8giF2EgT2bv8/ByHsVHFFLW+vT2dmXBj9w7xtXY4QXY6Erg52qrSKe97fRqivK/+5MQGnrjhw3mKG3INwYjNkbIWCo/XhSpsbb+vqawQq30iIHGl9HGW972EELsdWHm3XVtUHsabB7PTz8lzI3mk8ri6u37fneEi8BQbOAhePi/+zEKITeXN9GqXVJh64rK+tSxGiS5LQ1YFqTBZ+8/52SipNLLl3LP6eLrYuqW1UFRvh6sRmyLAGreoS4z2PQAgeAD3H1Icr3yijt8q3B7i1w1g2Z7f672oJUw2UZMC+L2DH+/DF3bD8DzDkl0YAi0iU05KiyyuurOWt9WlMHxzGgDAfW5cjRJckoasDPbFsH1uPFfLynAQGRXTSRk1ryD9ihKsTm4ygdeoAoI1TgSGDIO46o9cqaiQE9LL/wOLkYtQ54fcw/ndwbL0RvnZ9BNvegpDBkHAzDL3BOC0pRBf0lnUsl/RyCdF+JHR1kA83HeeDTce5Z1JvrhgaYetyWq6mArK21wesE5uh0rqogKsvRI2AwddA5AjoMRzcOmmYPE0piBlv3GY8C3uXGAHsu4fhh7/AgJmQcCv0ngwOjrauVog2UVJVy6J1aUwdFNp5DwiF6AQkdHWAbccKeGzpXib2C+YP0/rbupyz09oY0H46XJ3YBDl76sdhBfUzQkfkSIgaZTx36IJj0k5z84WkO4zbyX31vV/7vzJOjw6bA8NuggBZi050bm+vT6dEernsh6nGGMtq72cJRKtJ6GpnWmse+nQ34b7uvDR7GI72Oqtzyo9Gb07eYeO5sydEDofxDxoBKzIJPAJsW6MthQ6G6U/D5Y/DoW9hx3uw9jlY+0+InQgJt8DAK8HZ3daVCtEqpVW1vLkujcsHhsh8ge3JYoGKfCg7ab2danLf4HFVEfhGG1drD77GOIsgAaxLkNDVznaeKCItr5x/XDcUPw87HDhfdBxWPAwHv4bAPjDzOWMsVshgcJRfjzM4ucLgq41bcQbsXGwEsM9/bfSMxV1vjP8KHyaNpOgU3t1wjOLKWn57mSxmfVYWizE/oLna6IUyVxvT1phrrPe1YLJeNV3aNFRZH5fnnnn1NoCTO3iHglcoBPeD2AnGBUhZO2DTa7DhZfCJhEFXGbfIEV37DEMXJ/+rtrNlu7JxcXRg2uAwW5fSmKkafv6P0VujFFz2GIyZb4QK0TK+kTDpD8YA/GPrYPt7xinILW9AaBwk3GSEL68Q4+biZdsgZrEY/ymUZBm3Uut9RT5EJEC/6eBtZ7+nol2VVZv4X/JRLh0QQlxkN+jlslggP9WYLiZrhzG1TW1lfXA6I0xZ7y21rfse5Vj/7947DMKHGqHKK9T6eoPH52oXKguNnvX9X8GW/8HGV8A7oj6ARY2SANbJSOhqR2aL5uvdWVzSPxhfdzua2Tl1JXz7R6PxGTgLpv0d/KJsXVXn5eBgnGKMnQiV/4S9nxkBbMWfG2/n5F7fEHuFGjPle4UaE7OeboQ9g60NsWfrajDVQGm2cSvJhJLsxsGqxPpe0/88lCO4esG2t43nEQnQbwb0nw5hQ6W3rot7d0M6RRW1/LYrjuWyWIwJk7N2WG87IXtX/eTJTu4QMsAIPR6e4OhqXMnsaL05uTZ4reG9qzHe6ozXXIzJlr1CwT2gbcKQu7917OgcY2qeQytg/5ewdRFsetWYx3DgLCOARY+Wi3s6AQld7WhTWj6nSquZNcxOrlYszoDvHjGOmgJ6wU1LoO/ltq6qa3H3gxF3Gbf8I1CYbpxWqDvdcArKTxkTxh7faPQy0cwaki5eRvjyDGkc1DwCjKPfpqGqPPfMz3D2MBplnwhjnjSfCOMo2ScCfMKNiwE8g42pPk7tN46oD6+A1U/D6r8b7/ebZoSw2InG/GeiyyivNvG/tUe5pH8w8VF+ti7n4mhtDVg760NW9q76+QKd3CAsDobdaPQ+RyQYFwJ1piEUbr4Qf4NxqyqBw98ZAWz7O7D5NaN9GHglDLoaeo61zwBWkt0gBG832sGIBOh9KfSabMzd2MV1ot+4zmfZriw8XBy5bECobQsx1Rjd0mv+YTROly6AsQ/IqcT2Ftj7/MsKmU3GKb+GoazsZOOglnsI0pMbL5Hk7m+EIu9w4z8Rn4gzQ1Vr1poMHWzcJj5kfGfK90YI2/WxcVTt7GE0iv2mWU9D2vh3Wly09zYeo7Az9nJpbYxFrQtXO42wVVVkvO/oAqFDjPGVEQkQMcyYoLm1K1vYMzcfGHq9casuNf697vsSdnxgDG/wDLYGsKuMVTZsES7L86wheHv931VptvGecoSQgca8junrjKl5AIL6GwGs96UQM671Pf6dgISudlJjsvDt3hymDArF3cWGRxxHVxuzq+cdhgFXGKcS/Xvarh7RmKOTMeajJWOpTDXGHGluvu17laRXiHExQMLNxpJK6evg8LfGqY1D3xjbRCRC/xlGAAuLk9OQnUxFjdHLNbFfMAnR/rYup2VOHTQOHg98XT9XoIOzcbAw+GprwEqA4IHGab/uwtXbWD1jyC+hprw+gO36yDhg8giEnuPqx5CdHsLgGWIMbfAMufglzyqL6sfJZe2AzB1QfNz6poKgvsbatqf/jsLi6r9Ta6On/chPcGSVMSH1pleNv9vo0caciL0vhbD4jhm/Zq41pk7Sul3W4lVaN3Nqw4aSkpL01q1bbV3GRfvp4EnueHsrb96WxGUDbdArUJIF3z0K+z4H/1iY8Q/oN7Xj6xBdh9Zwcq8Rvg5/C5nbjNd9Io0esP4zIGbCBZ2GVEpt01ontXHFNtEZ2rD/rT3KU8sPsOTeMQzvacdTwWgNaWuNK/hSvjdOEw66GqJHGT28oYOlx/5saiog9QdjOEn2bmNYQ1Vx89u6eNWHsYbDGpoLaAA5u63hytqLVXCk/rP8Y41g1SPRGrCGtm7S7NoqOL6hPoSd3GO87h4AvS6x9oRNbvkyb01pbfTCFaZD0THjtHRhOhQeM24lGaAtRi/hr95t0Ue2pv2Snq52snRnFr7uzkzoG9yxX2yuhY2vwppnwWKCyY8apxIvIsBDAAAgAElEQVRlPI64WEoZR6hhccZVm6UnIeU7Y2zJrsWw9U1jfrfek40esP4zZdkkO1RZY+a1tUcY3yfIfgOXuRb2fg4b/mNM0OwZbLRlSXfK71RLuXjUX+V4mqnaOnThVIP7U1CWa70/BXkpkL6+vjfxXHwioUeCcaV2RIIRhC92PkdnN2vv1mTjedkp44zNkZ+M277PjdeD+tf3gvUcZ1wQdFpNuXEKujC9QaBKt4asdKitaPydXqHg19PoWfPvCf4xRqBvBxK62kFljZnv959kVnwELk4deDlvWjIsf8i4DLrfDGMyT5ktXbQX71BIvNW41VYZ485OD8Y/+DXc8L4xrkTYlQ82HSOvrIbfXm6HY7kqi4yB4RsXGheJBPWHK18y1j2VA8eL5+Rq9BC1pJfIXGv0CJ0OY6cDmsVknOqLsE6H0968QmDor4xb3anIVUYA2/Y2bFponIqMTDJqKzxm1NmQs6cRpPxjjN4yv571z/2iL/70aitI6GoHPx08RUWNmVnxHXTVYmkOfL8A9nxq/DLd+JFxqkeIjuLsBn2nGDf9vNE7EdjH1lWJJqpqzby29ihjewcyIsaOerkKjxn/eW5/F2rKjKtlr3wR+lwu81DZiqOz9SrncFtXUk+p+ot+xt5nHOyd2GgEsPT1RnjqN60+UJ2+eQTazbhTCV3tYOmuTEK8XRnVq527wc0m41LhVU8bk/hN+pOxbI8sRSNsSSljMkhhdz7cdJzc0mpevjHB1qUYMrYZpxD3f2VMXTLkl8YkzeHxtq5MdAbObkbPVa9LbFtHK7QodCmlpgMvAo7AG1rrZ5q8Hw28A/hZt/mz1nq5UioGOAAcsm66UWt9T9uUbp9KqmpZdSiXm0ZFt+86i8c3wte/g1P7oO9UmPGsMfeWEEI0o6rWzMI1RxjdK6D9DwjPxWIxLsT4+WU4/jO4+sKY+2DUPd1inibRvZ03dCmlHIFXgClABrBFKbVUa72/wWYLgE+01q8qpQYBy4EY63tHtNbD2rZs+/Xd3hxqTBaubM9TiweXwye3GnM0zf7QGLBsJ12nQgj79PGWE5wqrebfs23UHNdUwK4PYcN/javdfKNh2tOQeIsx7YEQ3UBLerpGAqla66MASqmPgKuAhqFLA6evCfUFstqyyM5k2e5sogLcSWivGZ4PfgOf3GacvrnlC2POJiGEOIdqk5lXVx9hZEwAYzq6l6sk25gvassbxhVxEYlw3VvG8jWdaUZ4IdpAS37jewAnGjzPAEY12eZx4Hul1P2AJ9BwbZlYpdQOoARYoLVOvvBy7Vt+WTXrU/O4e2IvVHv0PNUFrni45XMJXEKIFvlkywlySqp4/lfx7dM2NaS1cQX1wW/g0HLrfG7K6JEfex9Ej5GeedFttSR0Nfevo+mMqjcCb2utn1dKjQHeU0oNAbKBaK11vlJqOPClUmqw1rqk0RcoNQ+YBxAdHd3qH8JeLN+Tjdmi22etxQNfw6e3GXOh3LxEApcQdsSe27Bqk5n/rj5CUk9/xvZup14ui9kYZ3pouRG2CtOM1yMSjWXHBl/bLrN7C9HZtCR0ZQBRDZ5HcubpwzuB6QBa6w1KKTcgSGt9Cqi2vr5NKXUE6Ac0mq5Za/068DoYszlfwM9hF5buyqJviBf9Q9t4fMKBZfDp7dbA9XnrZvcVQrQ7e27DPt2aQXZxFf+4bmjb9nLVlBuX6h9cbszNVllgrHsYOxHG3m/0bNnTdANC2IGWhK4tQF+lVCyQCcwG5jTZ5jhwGfC2Umog4AbkKqWCgQKttVkp1QvoCxxts+rtSFZRJVvSC/n9lH5t27DtXwqfzZXAJYRotRqThVdXHyEx2o/xfYIu/gPLThkT4B5abswSbqoyet37ToMBM415tWRQvBBndd7QpbU2KaXuA77DmA5ikdZ6n1LqSWCr1nop8Hvgf0qpBzFOPd6utdZKqYnAk0opE2AG7tFat2Btgc7n691G51+bXrW4/yv47A6ji/7mJRK4hBCtsmR7BplFlfz92rgLPxjMPWwsdH5wOWRsAbRx5eHw243erJ5jjYk0hRDn1aJLR7TWyzGmgWj42l8aPN4PjGtmvyXAkoussVNYuiuL+EhfYoI82+YDTweuHsPhps8kcAkhWqXWbOGVVanER/kxsW8re7ly9sDuT4werfxU47XweLjkYaNHK3SIDIYX4gLI9bpt4GhuGXszS1jwi4Ft84H7v4JP5xprSd28RLrrhRCt9vn2DDIKK/nrVUNa18uVtQPemAJoiJlgTFraf0bL1usTQpyThK42sGxXNkrBFUPb4NTivi+NHi4JXEKIC1RrtvDyqlSGRvpySf/glu9YXWq0P14hMG91xyxoLEQ3IiuJXiStNUt3ZTIyJoAwX7eL+7B9X1gD1wgJXEKIC/bFjkxOFFTy28v6tq6Xa/kfoTAdrn1dApcQ7UBC10Xan13Ckdzyi5+ba+/n8NmdEDUSbv5MApcQ4oKYrGO54nr4cumAVgSn3Z8ay/RMeAhixrdfgUJ0YxK6LtLSXVk4OShmDLmI+Wj2LoEldxmB66ZPJXAJIS7YVzuzOJZfwQOt6eUqSIOvH4SoUTDpT+1boBDdmISui6C15utd2YzvG0SAp8uFfcjeJbDk10Zjd5P0cAkhLpzJOpZrULgPlw9sYS+XudY46FMOcO3/ZD1EIdqRhK6LsP14IZlFlcy60Lm59nxm7eEaZe3h8mrbAoUQ3cqy3Vmk5ZW3rpdr1d8hcytc+W/w79m+BQrRzUnoughLd2bh6uTA1MFhrd95z2fw+a+NxV8lcAkhLpLZovnPT6kMCPNm6qDQlu10dA2s+xck3AJDrm3fAoUQEroulMls4Zs92Vw2MAQv11Z2xzcMXHM+kcAlhLhoX+/O4mhuOb+9rC8ODi3o5SrPhy/uhsA+MOPZ9i9QCCHzdF2ojUcLyCur4crWzs21+1P4Yh5Ej4WbPgGXNprBXgjRbZktmpdWptA/1JtpLel51xq+mg8V+TDnY2mHhOgg0tN1gZbuysTL1YnJrbok+xMJXEKINrfpaD5Hcsv5zeTeLevl2vIGHP4WLn/CWN5HCNEhJHRdgGqTmW/35jB1cChuzo4t22nXx0ZXfs9xEriEEG1qbUoeTg6Kywa2YCxXzl747lHoMwVG39v+xQkh6kjougBrDuVSWmVq+VWLuz+FL+8xApd05Qsh2ti61FwSe/qff3xpTQUsuRPcfOHqV2XRaiE6mISuC7Bsdzb+Hs6M6xN0/o3LcmHZAxA1WgKXEKLN5ZdVszezhIl9W9AeffcI5B6EaxaCVyvWZBRCtAkJXa1UUWPix/0nmRkXjrNjC/741r0ApiqY9R8JXEKINrcuNQ+A8X3PE6L2L4Vtb8HYB6DPZR1QmRCiKQldrfTD/pNU1ppbdmqxOBO2vAnxcyCoT/sXJ4TodpJT8vB1dyauh+/ZNyrOgKX3Q0QCXPr/Oq44IUQjErpaadmuLMJ83BgRE3D+jdf+E7QFJv2x/QsTQnQ7WmvWpeQxvk8Qjme7atFihs/nGcv9/PJNcLrAJcuEEBdNQlcrFFfUsuZwLlcMDT//ZdkFabDjPRh+myytIYRoF6mnysgpqWLCucZzJT8Px9bDL56HwN4dV5wQ4gwSulphxb5sas2aWcNacGpxzT/AwQkmPNT+hQkhuqW1KafHc50ldB3fCKufhrjrIX52B1YmhGiOhK5WWLori5hAj3OPnQDIPQS7P4IRd4FPeMcUJ4TodpJTcukV5Emkv8eZb1YWwZK7wDcKfvGCTA8hhB2Q0NVCp0qr2HAkn1nxEajzNV6rnwYndxj/YMcUJ4TodqpNZjYdLWj+1KLWsOy3UJoN1y0CN5+OL1AIcQYJXS20fHc2Fg1Xnu+qxezdsO8LY6ZnzxbMmyOEEBdg27FCKmvNTGhuqogd78H+L2HyoxCZ1PHFCSGa1aLQpZSarpQ6pJRKVUr9uZn3o5VSq5RSO5RSu5VSMxu897B1v0NKqWltWXxHWroriwFh3vQN9T73hqv+bsz2PPb+jilMCNEtJVuX/hndO7DxG7mH4ds/QexEGPd/tilOCNGs84YupZQj8AowAxgE3KiUGtRkswXAJ1rrBGA28F/rvoOszwcD04H/Wj+vUzlRUMH240XnH0CfsdVYRHbs/eDu1zHFCSG6peSUXBKjmyz9Y6qGJXeAkxtc8zo4yMkMIexJS/5FjgRStdZHtdY1wEfAVU220cDpQQO+QJb18VXAR1rraq11GpBq/bxOZdlu48e5cuh5QtdPfwWPQBh1TwdUJYTorvLLqtmXVXLmeK4fH4ecPXD1f+UiHiHsUEtCVw/gRIPnGdbXGnocuFkplQEsB06fW2vJvnZv2a5sEqL9iApo5gqh09KS4ehqGP87cD3PKUghhLgI64/kozVM6NdgPNfh72Hjf2HkPOg/w3bFCSHOqiWhq7lL9XST5zcCb2utI4GZwHtKKYcW7otSap5SaqtSamtubm4LSuo4qadKOZBdcu5lf7SGVU+BdziMuLPjihNC2IWObsOSD+c2XvqnshC+vBdCBsOUv7b79wshLkxLQlcGENXgeST1pw9PuxP4BEBrvQFwA4JauC9a69e11kla66Tg4PMs2trBlu7MwkHBL4aeo6s+dSUc3wATHwJn944rTghhFzqyDdNak5ySx7g+gfVL/6SuhIo8uOIFcHZr1+8XQly4loSuLUBfpVSsUsoFY2D80ibbHAcuA1BKDcQIXbnW7WYrpVyVUrFAX2BzWxXf3rTWLN2VxZjegYR4n6Uh09oYy+UbDQm3dmyBQohu50ju6aV/GoS7tLXg6guRI2xXmBDivJzOt4HW2qSUug/4DnAEFmmt9ymlngS2aq2XAr8H/qeUehDj9OHtWmsN7FNKfQLsB0zAfK21ub1+mLa2N7OE9PwK7r3kHOuVHfwasnfCVa/IQrJCiHa39rB16Z8+DQbRp62FmHHg0OkuDheiWzlv6ALQWi/HGCDf8LW/NHi8Hxh3ln2fAp66iBptZumuTJwdFdMHn+XUosUMPz0FgX1gqKxrJoRof6eX/qm7sKfoBBSmwai7bVuYEOK8ZBKXs7BYNF/vzmZSv2B8PZyb32jfF5B7AC55GBxblF+FEOKCVZvMbDxa0HiB6/Rk4z5mgm2KEkK0mISus9iSXkB2cdXZl/0xm4zZ50MGw+BrO7Y4IUS3tP1Y0ZlL/6StNeYHDGk6Z7UQwt5I98xZLNudhbuzI1MGhTa/wa7FUHAEZn8osz4LITpEckqusfRPrwDjBa2t47kmSDskRCcg/0qbUWu2sHxPDpcNDMHDpZlcaqqGNc9CRCL0n3nm+0II0Q6SU/JIjPbH28065KHgKJRkQqycWhSiM5DQ1Yz1qXkUlNecfULU7e9C8Qm4dAGo5uZ/FUKItlVQXsPerOLG47nS1hr3sZNsU5QQolUkdDVj6a4sfNycmNS/mUkOaypg7T8heiz0vrTjixNCdEvrU/OMpX+ahi7vcOMKaiGE3ZPQ1URVrZnv951k+pAwXJ2amfNmyxtQdlJ6uYQQHSo5JRcfNyeGRvoZL2htXLkYM0HaIiE6CQldTaw+dIqyalPzVy1Wl8K6fxk9XDHNTksmhBBt7vTSP+P7BtUv/ZN7EMpzIXaibYsTQrSYhK4mvtiRSZCXC2N6BZ755saFUFkAkxd0fGFCiG7rSG4Z2cVVjO/TZKoIkNAlRCcioauBzWkFfLfvJDeMiMLJsckfTWUh/Pwf6P8LiBxumwKFEN1Scoqx9M8Z47n8osG/p42qEkK0loQuq2qTmYc/302kvzvzJzczKPXn/0B1MUx+pOOLE0J0a8kpecQ2XPrHYob0ddLLJUQnI6HLauHqoxzJLedvVw85c26uslzj1OLgayFsiG0KFEJ0S9UmMxuO5Dfu5crZA1VFMlWEEJ2MhC6M8RKvrErlyvgILukfcuYG6/4Fpkrp5RJCdLjTS/+M7yPrLQrR2XX70KW15tEv9uDm7MBfrmhm7bKSLGOaiPgbIahvxxcohOjW1qXm4uigGNO7wcU9aWshsC/4hNuuMCFEq3X70PXptgw2Hi3g4ZkDCfZ2PXODtf8EbYFJf+r44oQQ3Z6x9I9f/dI/5lo49rOM5xKiE+rWoSu/rJq/Lz/AiBh/bkiKOnODwnRjyZ/EW+UKISFEhysor2FPZjET+jaYKiJrJ9SUyXqLQnRC3Tp0/e2bA5RXm3j62jgcHJqZ0XnNP8DBCSY+1PHFCSG6vdNL/zReb3GNcS/juYTodLpt6EpOyeWLHZncO6k3fUK8z9wg9zDsWgwj7gKfsyx8LYQQ7WhdSp6x9E8P3/oX09ZC6BDwDDr7jkIIu9QtQ1dVrZkFX+6lV5Anv2luTi6A1U+DkzuM+7+OLU4IITi99E8u4/oE1U/WbKqGE5ukl0uITqpbhq6XVqZwLL+Cv10zBDfnZha1ztkL+z6H0feCV/CZ7wshRDs7kltOVnFV4/FcGVvAVCWD6IXopLpd6DqUU8rra49y3fBIxvY+S/f8qqfA1RfG3texxQkhhFVySi7QzNI/ygF6jrVRVUKIi9GtQpfFonnkiz14uznxyMyBzW+UsQ0OLYex94O7f8cWKIQQVutS8ogJ9Khf+gcgLRnC48Hdz3aFCSEuWItCl1JqulLqkFIqVSn152be/5dSaqf1dlgpVdTgPXOD95a2ZfGt9eHm42w7VsiCXwwiwNOl+Y1WPQUegTD6no4tTgghrGpMFjYczW98arGm3Di9KKcWhei0nM63gVLKEXgFmAJkAFuUUku11vtPb6O1frDB9vcDCQ0+olJrPaztSr4wp0qqeHbFQcb1CeTaxB7Nb5S5DY6shMsfB9dmrmgUQogOsP14IRU15sanFo9vBEuthC4hOrGW9HSNBFK11ke11jXAR8BV59j+RmBxWxTXlp5Ytp9qk4W/XR2HUs3MyQWQ/AK4+ULSnR1bnBBCNJCc0szSP+nJxryBUaNtV5gQ4qK0JHT1AE40eJ5hfe0MSqmeQCzwU4OX3ZRSW5VSG5VSV19wpRfhp4Mn+WZPNg9c2ofYIM/mNzp1AA5+DaPuATefji1QCCEaWJeSR0JUg6V/wBhE3yMJXL1sV5gQ4qK0JHQ11y2kz7LtbOAzrbW5wWvRWuskYA7wb6VU7zO+QKl51mC2NTc3twUltVx5tYn/9+U++oV6MW/iGV9db92/wNnTCF1CCNEKbdmGFZbXsLvp0j9VxZC1Q04tCtHJtSR0ZQANFyaMBLLOsu1smpxa1FpnWe+PAqtpPN7r9Dava62TtNZJwcFtOy/Wv344TGZRJX+/Jg4Xp7P8uAVpsOczSJoLHgFt+v1CiK6vLduw9UeMpX8m9GswnuvYBtAWWW9RiE6uJaFrC9BXKRWrlHLBCFZnXIWolOoP+AMbGrzmr5RytT4OAsYB+5vu2172ZhazaH0ac0ZFkxRzjjC1/kVwcIQxMi+XEMK2kg+fZekfR1eIHGm7woQQF+28Vy9qrU1KqfuA7wBHYJHWep9S6klgq9b6dAC7EfhIa93w1ONA4DWllAUj4D3T8KrH9mQyW3j48z0Eernyp+kDzr5hSTbs/ACG3QQ+4R1RmhBCNEtrzbrUPMb2brD0DxihK3oUOLvZrjghxEU7b+gC0FovB5Y3ee0vTZ4/3sx+PwNxF1HfBXtnwzH2ZBbz8pwEfN2dz77hhpfBYoZxv+244oQQohlH88rJLKrkN5MbjD+tKICTe2DyAtsVJoRoE11yRvrMokqe//4Qk/sH84u4c/ReVRTA1kUQdx0ExHZcgUII0Yzkw8Yg/IkNB9GnJxv3MoheiE6vy4UurTWPfbUXreHJq4acfU4ugI2vQm0FjP9dxxUohBBnkdzs0j9rjSureyTarjAhRJvocqFrxd4cfjxwigen9G3ccDVVVQKbX4MBV0DIOcZ8CSFEB6gxWdh4NJ/xDWehB2O9xZ5jwPEcwySEEJ1ClwpdJVW1PLZ0H4PCfbhj3HlOF25dZMx9M+H3HVOcEEKcw47jhZTXmBvPz1WaA3mH5NSiEF1EiwbSdxbPfXeIvLJq/ndrUuMrf5qqrYQNr0DvS6XLXghhF5JT8s5c+idNxnMJ0ZV0mZ6u7ccLeW/jMW4dE0N8lN+5N97xPpSfkl4uIYTdSE7JJSHKD5+GS/+krzXWgw0barvChBBtpkuErlqzhUc+30OYjxsPTet/7o3NtcZkqFGjoee4jilQCCHOoajCWPrnzPFca6HneGPyZiFEp9clQtcbyWkczCnliVmD8XI9zxnT3Z9A8Qmjl+tcVzYKIUQHWZ+abyz903A8V9FxKEyXU4tCdCGdPnQdz6/gxZWHmTY4lKmDw869scVsLGwdFgd9p3RMgUIIcR7JKbl4uzkRH9lw6Z/T47lkvUUhuopOHbq01jz65R6cHBx4YtaQ8+9wYCnkp0gvlxDCbmitSU7JY1xzS/94BEHwQNsVJ4RoU506dC3dlUVySh5/mNafMN/zrEmmNSQ/D4F9YeCsjilQCCHOI8269E+j8VxaG6ErdgI4dOpmWgjRQKf916y15tXVRxgW5cfNo3uef4fUHyFnD4x/UAalCiHsRnJKHtBk6Z+Co1CaBTFyalGIrqTTztOllOLjeWMorqzF0eE8pwq1hrXPgW8UDP1VxxTYjdTW1pKRkUFVVZWtSxF2zs3NjcjISJydZXb105JTcukZ6EF0YMOlf9YY97GTbFNUFyDtkmhrbdF+ddrQBeDr4YyvRwt++GM/w4mNMPM5WUqjHWRkZODt7U1MTMy517oU3ZrWmvz8fDIyMoiNlQXmwZjuZsORfK5J7NH4jbS14B0Bgb1tU1gXIO2SaEtt1X512tOLrZL8HHgGQ8LNtq6kS6qqqiIwMFAaNnFOSikCAwOl56GBHceLzlz6R2vjysXYCXLBz0WQdkm0pbZqv7p+6MrcDkd+gjHzwdnd1tV0WdKwiZaQ35PGklNyz1z659QBqMiT+bnagPy+ibbUFr9PXT90rXvBWEYj6U5bVyLakZeX1xmvHTp0iEsuuYRhw4YxcOBA5s2bx3fffcewYcMYNmwYXl5e9O/fn2HDhnHrrbeyevVqlFK8+eabdZ+xY8cOlFI899xzjT77qaeeqvscR0fHuscvvfRSi2vetGkTDz74YKt/1i1btqCUYuXKla3eV9iXtSl5DGu69E/aWuNeQlenlp6ezpAhjacyevzxxxu1JS+88AIDBgwgLi6O+Ph4fve731FbWwtATEwMcXFxDB06lEmTJnHs2LG6/ZRS/P739cvYPffcczz++ON13+Hh4cGpU6fq3m+ufRw1ahTDhg0jOjqa4ODgujYsPT29xT/jo48+yqpVq1q8/Wnz588nOjoarXWr9+3sunboOnUQDiyDkXeDm4+tqxEd7IEHHuDBBx9k586dHDhwgPvvv59p06axc+dOdu7cSVJSEh988AE7d+7k3XffBSAuLo6PP/647jM++ugj4uPjz/jsRx99tO5z3N3d6x4/8MADjbYzmUxnrW/UqFH861//avXPtXjxYsaPH8/ixYtbvW9rnKt2cfGKKmrYnVHEhKZL/6Qng19P8Iu2TWGiQyxcuJDvv/+ejRs3smfPHrZs2UJISAiVlZV126xatYrdu3dzySWX8Le//a3udVdXVz7//HPy8vKa/eygoCCef/75c37/pk2b2LlzJ08++SQ33HBDXRsWExPTaDuz2XzWz3jqqaeYPHlyC37axp+3dOlSwsPDWb9+fav2bQ2tNRaLpd0+/0J17dC17l/g7Amj77V1JcIGsrOziYyMrHseFxd33n2io6Opqqri5MmTaK1ZsWIFM2bMaNX33nzzzfz+979n8uTJPPLII2zcuJExY8aQkJDAuHHjSElJAeDHH3/k6quvBmDBggXceeedTJo0iV69evHKK680+9kWi4UlS5bwzjvv8O2331JTU1P33ltvvcXQoUOJj49n7ty5AOTk5HDVVVfVvb5p0yZSU1MZNmxY3X7PPPNMXYM+fvx4Hn30USZOnMjLL7/MV199xahRo0hISGDq1Kl1R8+lpaXcdtttdUfiX375Ja+99hp/+MMf6j731Vdf5Y9//GOr/uy6k5+PNLP0j8VshC7p5erynnrqKV599VX8/PwAcHFx4c9//jM+Pmd2EIwZM4bMzMy6505OTsybN++sB2133HEHH3/8MQUFBa2uy2Qy4efnx4IFCxg5ciSbN2/mscceY8SIEQwZMoR77rmnrofq5ptv5ssvvwQgMjKSxx9/nISEBIYOHcrhw4eb/fwff/yRhIQE5s2b1+jAsbk2BeCbb74hMTGR+Ph4pk6dChjt5b///e+6fQcMGEBGRgapqal1NSYmJpKdnc28efNISkpi8ODBPPnkk3X7bNq0iTFjxhAfH8+oUaOoqKhg7Nix7N27t26bUaNGsW/fvlb/GZ5Lp7568ZwK0mDPp0bg8giwdTXdxhPL9rE/q6RNP3NQhA+PXTm41fs9+OCDXHrppYwdO5apU6cyd+7cugbuXK677jo+/fRTEhISSExMxNXVtdXffeTIEVauXImDgwPFxcWsW7cOR0dHVqxYwYIFCxr1pp12+PBhVq5cSVFREQMHDuSee+7B0bHxnHJr165lwIAB9OrVi3HjxrFixQpmzZrFrl27ePbZZ/n5558JCAioa2znz5/PlClTuO+++zCZTFRUVDQ67dCckpIS1q41TnEVFhYya9YslFIsXLiQ559/nmeffZbHH3+c4OBg9uzZg9aaoqIinJycGDZsGE8//TROTk689dZbvP32263+s+sufNycmTootPHSPzm7oapYpopoY/bULoERMMrKylp8FdyKFSvqDtBOmz9/PkOHDm32wMbLy4s77riDF198kSeeeKLV9RUXF5OYmFh3MNa/f3+eeOIJtNbMmTPnrAejoaGh7Nixg5deeokXXniBhQsXnrHN4sWLueFJlW4AABupSURBVPHGG5kxYwaPPfYYL774Ik5OTs22KTk5Odx7770kJyfTs2fPFoXI/fv389Zbb9V99zPPPENAQAAmk4nJkydz3XXX0atXL2bPns2SJUtITEykuLgYV1dX7rzzTt5++22ee+459u/fD8DgwRf2d3w2Xben6+eXjElQx9xn60qEjcydO5cDBw5w/fXXs3r1akaPHk11dfV59/vVr37Fp59+Wtc4XIjrr78eB+tM4kVFRVx77bUMGTKEhx566KxHTldccQUuLi6EhIQQEBBAbm7uGdssXryY2bNnAzB79uy6I8WffvqJG264gYAA4wDj9P3q1au5++67AePouLmj6KZOfz7A8ePHmTp1KnFxcbzwwgt1tf/444/Mnz8fMMaX+Pv74+3tzcSJE/n222/Zt28fjo6ODBo06Px/WN3U+L5BvH5rUpOlf2S9xa7ibIOulVJorRu9f3qsaUxMDD///HPd65MnTyYkJIQff/yROXPmNPocHx8fbr311rOOI33ggQd45513KClpfdh0cXHhmmuuqXu+cuVKRo4cSXx8PGvWrDlrG3bttdcCMHz48GbHhlVXV/P9998za9Ys/Pz8SExMrBub2lybsmHDBiZPnkzPnsYE6KfbtXPp3bs3I0aMqHu+ePFiEhMTSUxM5MCBA+zfv58DBw4QHR1NYmIiAL6+vjg6OjJ79my++uorTCYTixYtqjtj0Ja6Zk9XaQ7seB+G3QQ+4bauplu50CO/9hIREcEdd9zBHXfcwZAhQ9i7dy/Dhw8/5z5hYWE4Ozvzww8/8OKLLzZqBFvK09Oz7vGjjz7KtGnT+M1vfkNqairTp09vdp+GPWqOjo5njKmqra3liy++YPny5TzxxBNYLBaKioooLy8/oxFvqOnrTk5OjcY6VFVV4eRU3xQ0rH3+/Pk88sgjzJw5kx9//JFnnnkG4Kzfd9ddd/HCCy8QExPTLg1Wl5e2FoL6gXeYrSvpUmzRLgUGBlJYWNjotYKCAmJjY/Hx8cHT05O0tDRiY2OZNm0a06ZN44orrmg0ZGDVqlV4enpy++2385e//IUXXnih0ef93//9H4mJic3+W/Pz82POnP/f3r3HRVnm/x9/XYoGnkI0D0EudNLcgRkIRcGfB74u2IanXRPR0tW04pHrevi56uZjNX/qI3+a5S9dtlbT2K9CmbbZqng23TVPIZqHzE1JTcUCBQUpxev3x8D9BRlggIG5wc/z8eDhzD0z1/1hhDfXXPd139dw/vKXv1S6di8vL+P3Oy8vj/Hjx5Oamoqvry8zZ84s87IJRRnmKL/AfqgwOzvbGD3Kzc3Fx8eH6Ohoh5lSVs44yrAixfPrzJkzLFmyhIMHD+Lt7c3zzz9Pfn5+me02bdqU3r17s2HDBtatW0daWlqZ71FVOTXSpZTqp5Q6rZT6j1JquoPH31JKpRV+faOUul7ssVFKqTOFX6NcWXyZ9r1jnxsR8Yda2Z0wp5SUFONMoCtXrpCZmYmvr28Fr7KbM2cOCxYsKHV4ryqys7ON/VbncNvWrVvp0qULFy5cID09nfPnz9O/f382bNhA3759SU5ONobfi/7t06ePMcxeUFBATk4O7dq149KlS1y7do38/Hw2btxYYe1aaz744ANje1RUFEuXLgXswVj0xyUiIoJvv/2WtWvXEhsbW+Xv9b5UcNt+IWeZz1UvNGvWjPbt2xsjOVlZWaSkpNCjRw8AZsyYQXx8PNev2/9caq0ddma8vLx4++23SUxMLHV4zcfHh6FDh5Y447q4yZMn8+6771brpJhbt27RoEEDWrduzY0bN1i3bl2V20pKSmLVqlWkp6eTnp7O2bNn2bx5M/n5+Q4zJSIigp07dxpnbhZ9//7+/nz55ZcAHDx4kAsXLjjcX05ODs2bN6dFixZcvnyZLVu2APZDht999x2pqanG84pOGBg7dizjx48nPDycBx980GG71VFhp0sp1RBYBjwDdAbilFIljhlorSdprW1aaxvwDrC+8LU+wCwgDOgKzFJKtXTtt3CPvCw4vBICh4CPXPX6fpGXl4efn5/xtXjxYrZu3YrFYsFqtRIdHc3ChQtp1865EYTw8PBScyiqatq0aUydOpWIiIhqtZOUlFRiyB/gt7/9LWvWrDHmdvTs2RObzWZMaF+6dClbtmwhMDCQ0NBQvv76azw9PfnTn/5Ely5dGDBgQLmHAGfPns3gwYPp1asXbdu2NbbPmjWLjIwMLBYLNpuNvXv3Go8NGTKEnj171khg1WuXjsDtXFlvsR5JTExk7ty52Gw2IiMjmTVrFo89Zl9lID4+nr59+xIWFkZQUBAREREEBwcTHBxcqp327dsTFxfn8ASbKVOmlHsW4+DBg52aVlGWVq1aMWrUKCwWC4MHDyYsLKxK7dy8eZMdO3aUmAvWvHlzwsLC2Lhxo8NMadu2LQkJCQwcOBCr1cqIESMA+/SNjIwMgoODWbFiBY8++qjDfYaEhNC5c2csFgvjxo0zMviBBx4gKSmJ+Ph4Y4J+0XsUFhZGkyZNam6kXmtd7hfQHdhS7P4MYEY5z98H/KrwdhzwbrHH3gXiytvf008/ratl5zytZ7XQOuNk9doRTjt5Ut5r8T+io6P17t27y3zc0c8LcFhXkEV15avKGfb5/7Vn180fq/Z6UYLkkqiK8+fP644dO+q7d+86fLy6+eXM4UVfoPjY3cXCbaUopX4BBAA7K/tal/jpBhz4K3SKgTZP1dhuhBClZWZm8uSTT9KyZUt69ZKz7yrt3B5oGwhNW1X8XCGEy61cuZLw8HDmz59fY6sZODOR3tGey7qM7DDgY6110dXUnHqtUuol4CWwXyepyg6/bz/d+n9NrnobQogqadWqVZnX5qnvqp1ht/PhwkEIHePiyoQQzho9enSNnwDkzEjXReCRYvf9gEtlPHcYUPwy2U69Vmv9ntY6VGsd+tBDD937sHNu34J9S+HRPuBb/tlpQgjhStXOsIuH4E6+TKIXop5zptN1CHhCKRWglGqMvWO14d4nKaU6Ai2BL4pt3gJEKaVaFk6gjyrc5npH/htyr0LP/10jzQshRI1J3wuqAfwi3N2VCCFqUIWHF7XWd5RS47F3lhoC72utTyil5mCfPFbUAYsDkgsnlRW9Nksp9X+wd9wA5mitK78uQUUKbsO//x88Ega/qN4ZYkIIUevO7YH2NvCUMz6FqM+cujiq1noTsOmebX++5/7sMl77PvB+FetzzldrIfs8PLsIamjymxBC1Iifc+HiYej+qrsrEULUsLq/DNDdAti72H7WzxNR7q5GuEmzZs1KbTt9+jS9e/fGZrPx1FNP8dJLLxnLbdhsNpo1a0bHjh2x2WyMHDmS3bt3o5QqcaHBI0eOoJRi0aJFJdqeN2+e0U7Dhg2N22UtyVGWs2fPkpycXO5zFi5cSJMmTbhx40al2hZ1xPn9cPe2zOeqZ9LT07FYLCW2zZ49u0SWLF68mE6dOhEYGIjVamXy5MnGBZ39/f2NxZ979eplXCAU7KtMTJkyxbi/aNEiZs+ebeyjSZMmJdZYdZSPYWFh2Gw2OnTowEMPPWRkmKPle8qzfv16vv7663KfY7FYeOGFFyrVbn1V9ztdpz6DzDP2MxZllEsUM2HCBCZNmkRaWhqnTp3i97//PdHR0aSlpZGWlkZoaCirV68mLS2NxMREAAIDA0ssRp2cnIzVai3V9muvvWa04+XlZdyeMGFCpWp0ptOVlJTE008/zaefflqptiur6IrMopad2wMNGkGHbu6uRNSiv/71r2zdupX9+/fz1VdfcejQIdq0acOtW7eM5+zatYtjx47Ru3dvY/FpsF/cc/369eVeFPXNN98sd/8HDhwgLS2NOXPmEBsba2SYv79/pb6Pijpdx44dw8PDg507d5b43lytOlfdr011u9OlNex9E1o9Dp0HursaYTKXL1/Gz8/PuB8YGFjhazp06EB+fj4ZGRlorUlJSSlxBWVnZGRk8Jvf/IbQ0FC6du3K/v37Afui1FarFZvNRkhICLm5uUyfPp1du3aVOUp2+vRpCgoKmD17trG4NdgDZtKkSVgsFoKCgoz11Q4cOED37t2xWq2EhYWRl5fH8uXLmThxovHafv368a9//Ys7d+7g7e3NzJkz6dq1KwcPHmTWrFl06dIFi8XCK6+8UnRRY7755hsiIyOxWq2EhISQnp5OXFxciSWEYmNj2bSpxCwE4Yxze8AvFBo3rfi5ot6YN28eCQkJeHt7A/ZFpqdPn+5wUfru3bvz/fffG/c9PDx46aWXeOuttxy2PWbMGD788MNSywY5a/PmzXTv3p2QkBBiY2PJzc0FYOrUqXTu3JmgoCCmTZvG3r172bRpE5MmTSpzlCwpKYmRI0cSGRnJP//5T2O7o0wBmD9/vjHy99prrwHQo0cPYx3EK1eu8PjjjwOwfPlyhg0bRkxMDM888ww5OTlERkYSEhJCUFBQif2tXLmSoKAgrFYro0eP5vr16zz66KNGZ+369esEBATU+IfPur3g9X92wJVjMHAZNKj+GnnCBTZPhytfubbNdoHwzBuVftmkSZOIjIwkPDycqKgoRo8ebQRceYYMGcLatWsJDg4mJCSkxELUzpgwYQJ//OMf6datG+np6cTExHD8+HEWLlzIe++9R1hYGDdv3sTT05M33niDpUuX8o9//MNhW0lJSQwbNow+ffowevRoMjMzadWqFQkJCVy6dImjR4/SsGFDsrKyyM/PZ9iwYaxbt46QkBCys7MrrD07O5uQkBDjU3THjh15/fXX0VozfPhwo9MZFxfH7Nmz6d+/P/n5+dy9e5exY8eSkJDAs88+y7Vr1zh06BBr1qyp1Ht138vPhstp0HOquyup30yUSwA3btzg5s2bBAQ4t1RdSkpKqWXJXn31VWP5r3s1a9aMMWPGsGTJEl5//fVK1Xb16lXeeOMNduzYQZMmTZg3bx5LlizhxRdfZNOmTZw4cQKlFNevX8fb25tf//rXDBkypMxl0z766CP27NlDp06dWL58Oc899xyAw0z57LPP2Lx5MwcPHsTLy8upTuMXX3xBWloaLVu25Pbt23z66ac0b96cq1evEhERQUxMDEePHmXBggXs27cPHx8fsrKy8Pb2JiIigpSUFGJiYlizZg1Dhw51yXq75anbI117F0ELPwgc6u5KhAmNHj2aU6dO8dxzz7F79266devm1BpkQ4cOZe3atSQlJREXF1fp/W7fvp1XXnkFm83GoEGDuHbtGrdu3SIiIoKJEyfyzjvvkJOT49Qvd3JyMsOGDaNBgwYMGjSIjz/+uMQ+itrw8fHh1KlTdOjQgZCQEAAefPDBCvfRuHHjEus57tixg65du2K1Wvn88885ceIE165d48cff6R///4AeHp60qRJEyIjIzl58iSZmZmsXr26VgKr3vluH+i7st5iPVTWFc2VUmitSzxeNNfU39+fffv2Gdv79OlDmzZt2L59O8OHDy/RTosWLRg5cmSZ80gnTJjABx98QE5OTqXq3rdvHydPniQ8PBybzcbq1atJT0/Hx8eHBg0aMG7cOD755BOaNq14ZPaLL77Az88PX19ffvWrX3HgwAGys7PLzJTt27czZswYvLy8AHuuVSQqKoqWLe1LOmutmTZtGkFBQURFRXHhwgV+/PFHdu7cSWxsrNFe0b9jx45l5cqVgH0krKYvjAp1eaRLa+gxCe78BB6N3V2NKFLFT3415eGHH2bMmDGMGTMGi8XC8ePHefrp8i+e265dOxo1asS2bdtYsmRJiRB0htaagwcP0rhxyZ/LmTNnMmDAADZu3EiXLl3YvXt3ue2kpqZy7tw5+vTpA8BPP/3EsWPHePnll0uFdtF+HQW9h4cHd+/eNe7n5+cbt728vIzX5OXlMX78eFJTU/H19WXmzJnGcx21q5RixIgRrFmzhlWrVskoV1Wc2wMenuDXxd2V1G9uyKVWrVpx7dq1EtuysrIICAigRYsWNG3alHPnzhEQEEB0dDTR0dHExMTw888/G8/ftWsXTZs25Xe/+x1//vOfWbx4cYn2Jk6cSEhIiMPOgre3N8OHDzemHjhLa02/fv34+9//Xuqxw4cPs23bNpKTk0lISGDr1q3ltpWUlMTx48eNeWI5OTl88sknDBw40GGmOJNhxfMLKNH5S0xMJDs7m9TUVDw8PPDz8yM/P7/Mdnv16sX48ePZtWsXjRo1olOnTuV+P65Qd0e6lIIno6HzAHdXIkwqJSXFOBPoypUrZGZm4uvr3NKfc+bMYcGCBVUauenbty/Lli0z7hfNRfj2228JCgpixowZBAcHc/r0aZo3b17mWYlJSUnMnTuX9PR00tPTuXTpEmfPnuX7778nKiqKhIQEY/5BVlYWv/zlL/nuu+9ITU0F7AFXUFCAv78/R44cQWtNeno6X375pcP93bp1iwYNGtC6dWtu3LjBunXrAGjZsiWtW7fms88+A+yhl5eXB9hHExcuXIinpycdO3as9Ht13zu31359wUae7q5EuFizZs1o3749O3bsAOy/oykpKfTo0QOAGTNmEB8fz/Xr1wF7h+PeDgXYPxi9/fbbJCYmljrc5uPjw9ChQ0uccV3c5MmTeffddys1yTw8PJzPP/+cs2fPApCbm8uZM2e4ceMGOTk5xMTE8NZbb3HkyBGAMjOsoKCAdevWcfLkSSPD1q9fT1JSUpmZEhUVxYoVK4wJ90Xfr7+/v5FbRaP9jmRnZ9OmTRs8PDzYtm2bMQ+ub9++JCcnG+0Vfx+ff/55RowYUSujXFCXO11CFJOXl4efn5/xtXjxYrZu3YrFYsFqtRIdHc3ChQtp166dU+2Fh4eXOUehIsuWLePf//43QUFBdO7cmb/97W+A/bTuoonv3t7eREVFERwcTEFBAVartcRhAq01H374YYlDf0opBg0aRHJyMi+//DLt2rUzJoZ+9NFHPPDAAyQlJREfH4/VaiUqKoqffvqJXr164evrS2BgINOnT8dmszmsu1WrVowaNQqLxcLgwYMJCwszHlu9ejVvvvkmQUFB9OjRgx9++AGwjyQ++eSTtRZY9UpuJmR8BQFyaLG+SkxMZO7cudhsNiIjI5k1axaPPfYYAPHx8fTt25ewsDCCgoKIiIggODiY4ODgUu20b9+euLi4Eh/mikyZMqXcsxgHDx7s1LSKIm3btmXFihXExsZitVoJDw/nm2++ITs7m2effRar1UpkZKQx6hYXF8f8+fNLTaTftWsXAQEBtG3b1tjWp08f0tLSyMjIcJgpMTEx9OvXj9DQUGw2m3GiwNSpU1myZAnh4eGlRg+Le+GFF9i3bx+hoaGsXbuWJ554AsCY+9azZ09sNhtTp/7PHMoRI0aQnZ1NbGys0+9RdahiF5A3hdDQUH348GF3lyEq4dSpUzz11FPuLkO4QW5uLoGBgRw9epTmzZs79RpHPy9KqS+11qE1UWNtczrDTvwD1o6CF7fBI11rvrD7jOSScEZycjJbtmwx5nZVpLr5VXfndAkh3GrLli2MGzeOqVOnOt3hEsU80AyeiIaHS49sCCFqXnx8PNu3byclJaXW9imdLiFElURHR3P+/Hl3l1F3Pd7X/iWEcIuEhIRa36fM6RJCCCGEqAXS6RIuYba5gcKc5OdE1Cb5eROu5IqfJ+l0iWrz9PQkMzNTAk6US2tNZmYmnp5yeQRR8ySXhCu5Kr9kTpeoNj8/Py5evGhcRkCIsnh6epZYD1OImiK5JFzNFfklnS5RbY0aNXJ6DTEhhKgNkkvCjOTwohBCCCFELZBOlxBCCCFELZBOlxBCCCFELTDdMkBKqR+A7yrxktaA44WnzEXqdC2p07XcXecvtNYPuXH/LlPJDHP3+14es9Zm1rrAvLVJXZVXmdqczi/TdboqSyl1uC6s2SZ1upbU6Vp1pc76xszvu1lrM2tdYN7apK7Kq6na5PCiEEIIIUQtkE6XEEIIIUQtqA+drvfcXYCTpE7Xkjpdq67UWd+Y+X03a21mrQvMW5vUVXk1Uludn9MlhBBCCFEX1IeRLiGEEEII06uznS6lVD+l1Gml1H+UUtPdXY8jSqlHlFK7lFKnlFInlFJ/cHdN5VFKNVRKHVFK/dPdtZRFKeWtlPpYKfV14fva3d01OaKUmlT4f35cKZWklDLFKs9KqfeVUleVUseLbfNRSm1TSp0p/LelO2u8X5gxw+pCZpkxp8ycS2bJIjNnTxm1LSz8/zymlPpEKeXtin3VyU6XUqohsAx4BugMxCmlOru3KofuAFO01k8B3YBXTVpnkT8Ap9xdRAWWACla606AFRPWq5TyBSYAoVprC9AQGObeqgyrgH73bJsO7NBaPwHsKLwvapCJM6wuZJYZc8qUuWSyLFqFebNnFaVr2wZYtNZBwDfADFfsqE52uoCuwH+01me11j8DycBAN9dUitb6stY6tfD2Dey/iL7urcoxpZQf8Cyw3N21lEUp1QLoCawA0Fr/rLW+7t6qyuQBeCmlPIAmwCU31wOA1noPkHXP5oHAB4W3PwAG1WpR9ydTZpjZM8uMOVUHcskUWWTm7HFUm9Z6q9b6TuHd/YCfK/ZVVztdvsCFYvcvYqJgcEQp5Q8EAwfcW0mZ3gb+CNx1dyHleBT4AVhZeHhhuVKqqbuLupfW+ntgEXAeuAxka623ureqcrXVWl8G+x9doI2b67kfmD7DTJpZZswp0+ZSHciiupI9Y4DNrmiorna6lINtpj0NUynVDFgHTNRa57i7nnsppWKAq1rrL91dSwU8gBAgQWsdDORiwkNhhfMSBgIBwMNAU6XU8+6tSpiMqTPMjJll4pwybS5JFlWfUuo17IfdV7uivbra6boIPFLsvh8mOXxzL6VUI+zhtVprvd7d9ZQhAhiglErHfpgjUin13+4tyaGLwEWtddEn74+xh53Z9AXOaa1/0FrfBtYD4W6uqTwZSqn2AIX/XnVzPfcD02aYiTPLrDll5lwyexaZOnuUUqOAGGCEdtH1tepqp+sQ8IRSKkAp1Rj7xMANbq6pFKWUwn6c/5TWerG76ymL1nqG1tpPa+2P/b3cqbU23achrfUV4IJSqmPhpv8CTrqxpLKcB7oppZoU/gz8FyaZWFuGDcCowtujgE/dWMv9wpQZZubMMmtOmTyXzJ5Fps0epVQ/YBowQGud56p2PVzVUG3SWt9RSo0HtmA/G+N9rfUJN5flSATwAvCVUiqtcNuftNab3FhTXfd7YHXhH6qzwGg311OK1vqAUupjIBX7sPQRTHLlZaVUEtAbaK2UugjMAt4APlJKvYg9pJ9zX4X3BxNnmGRW1Zgyl8yURWbOnjJqmwE8AGyz91fZr7V+pdr7kivSCyGEEELUvLp6eFEIIYQQok6RTpcQQgghRC2QTpcQQgghRC2QTpcQQgghRC2QTpcQQgghRC2QTpcQQgghRC2QTpcQQgghRC2QTpcQQgghRC34/5P7Hgin3EkxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc037349d68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "f, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(10, 4))\n",
    "ax1.plot(range(len(lstm_model.history['train_acc'])), lstm_model.history['train_acc'], \n",
    "         label='LSTM Train Accuracy');\n",
    "ax1.plot(range(len(lstm_model.history['eval_acc'])), lstm_model.history['eval_acc'], \n",
    "         label='LSTM Test Accuracy');\n",
    "ax2.plot(range(len(ugrnn_model.history['train_acc'])), ugrnn_model.history['train_acc'],\n",
    "         label='UGRNN Train Accuracy');\n",
    "ax2.plot(range(len(ugrnn_model.history['eval_acc'])), ugrnn_model.history['eval_acc'],\n",
    "         label='UGRNN Test Accuracy');\n",
    "ax1.legend();\n",
    "ax2.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test network on new reviews\n",
    "---\n",
    "\n",
    "I think it will be nice to test the network on new reviews that appeared recently on IMDb. I have selected three reviews posted on February 2018 for the movie Bad Apples. Feel free to play with new reviews or invent a new one yourself! The network mananaged to identify the sentiment correctly in all three cases so I am pretty impressed :)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from models_checkpoints/ImdbRNN/-0\n"
     ]
    }
   ],
   "source": [
    "################################################################\n",
    "# Restore trained model\n",
    "################################################################\n",
    "tf.reset_default_graph()\n",
    "checkpoint_directory = 'models_checkpoints/ImdbRNN/'\n",
    "device = 'gpu:0' if tfe.num_gpus()>0 else 'cpu:0'\n",
    "lstm_model = RNNModel(vocabulary_size=len(word2idx), device=device, \n",
    "                      checkpoint_directory=checkpoint_directory)\n",
    "lstm_model.restore_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "###############################################################\n",
    "# Import/download necessary libraries to process new sequences\n",
    "###############################################################\n",
    "import nltk\n",
    "try:\n",
    "    nltk.data.find('tokenizers/punkt')\n",
    "except LookupError:\n",
    "    nltk.download('punkt')\n",
    "from nltk.tokenize import word_tokenize\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_new_review(review):\n",
    "    '''Function to process a new review.\n",
    "       Args:\n",
    "           review: original text review, string.\n",
    "       Returns:\n",
    "           indexed_review: sequence of integers, words correspondence \n",
    "                           from word2idx.\n",
    "           seq_length: the length of the review.\n",
    "    '''\n",
    "    indexed_review = re.sub(r'<[^>]+>', ' ', review)\n",
    "    indexed_review = word_tokenize(indexed_review)\n",
    "    indexed_review = [word2idx[word] if word in list(word2idx.keys()) else \n",
    "                      word2idx['Unknown_token'] for word in indexed_review]\n",
    "    indexed_review = indexed_review + [word2idx['End_token']]\n",
    "    seq_length = len(indexed_review)    \n",
    "    return indexed_review, seq_length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "sent_dict = {0: 'negative', 1: 'positive'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "review_score_10 = \"I think Bad Apples is a great time and I recommend! I enjoyed the opening, which gave way for the rest of the movie to occur. The main couple was very likable and I believed all of their interactions. They had great onscreen chemistry and made me laugh quite a few times! Keeping the girls in the masks but seeing them in action was something I loved. It kept a mystery to them throughout. I think the dialogue was great. The kills were fun. And the special surprise gore effect at the end was AWESOME!! I won't spoil that part ;) I also enjoyed how the movie wrapped up. It gave a very urban legends type feel of \\\"did you ever hear the story...\\\". Plus is leaves the door open for another film which I wouldn't mind at all. Long story short, I think if you take the film for what it is; a fun little horror flick, then you won't be disappointed! HaPpY eArLy HaLLoWeEn!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "review_score_4 = \"A young couple comes to a small town, where the husband get a job working in a hospital. The wife which you instantly hate or dislike works home, at the same time a horrible murders takes place in this small town by two masked killers. Bad Apples is just your tipical B-horror movie with average acting (I give them that. Altough you may get the idea that some of the actors are crazy-convervative Christians), but the script is just bad, and that's what destroys the film.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "review_score_1 = \"When you first start watching this movie, you can tell its going to be a painful ride. the audio is poor...the attacks by the \\\"girls\\\" are like going back in time, to watching the old rocky films, were blows never touched. the editing is poor with it aswell, example the actress in is the bath when her husband comes home, clearly you see her wearing a flesh coloured bra in the bath. no hints or spoilers, just wait till you find it in a bargain basket of cheap dvds in a couple of weeks\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_reviews = [review_score_10, review_score_4, review_score_1]\n",
    "scores = [10, 4, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The sentiment for the review with score 10 was found to be positive\n",
      "The sentiment for the review with score 4 was found to be negative\n",
      "The sentiment for the review with score 1 was found to be negative\n"
     ]
    }
   ],
   "source": [
    "with tf.device(device):\n",
    "    for original_review, score in zip(new_reviews, scores):\n",
    "        indexed_review, seq_length = process_new_review(original_review)\n",
    "        indexed_review = tf.reshape(tf.constant(indexed_review), (1,-1))\n",
    "        seq_length = tf.reshape(tf.constant(seq_length), (1,))\n",
    "        logits = lstm_model.predict(indexed_review, seq_length, False)\n",
    "        pred = tf.argmax(logits, axis=1).numpy()[0]\n",
    "        print('The sentiment for the review with score %d was found to be %s'\n",
    "              %(score, sent_dict[pred]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing RNN cell activations\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The part of this tutorial has been inspired by the work of Karpathy in \"Visualizing and understanding recurrent neural networks\" ([link here](https://arxiv.org/abs/1506.02078)).\n",
    "\n",
    "We will use the library seaborn for plotting the heatmap. You can get it by typing in the terminal:\n",
    "> **pip install seaborn**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries for RNN visualization\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def VisualizeRNN(model, X):\n",
    "    ''' Function to return the tanh of the cell state at each timestep.\n",
    "        \n",
    "        Args:\n",
    "            model: trained RNN model.\n",
    "            X: indexed review of shape (1, sequence_length).\n",
    "            \n",
    "        Returns:\n",
    "            tanh(cell_states): the tanh of the memory cell at each timestep.       \n",
    "    '''\n",
    "    \n",
    "    # Initialize LSTM cell state with zeros\n",
    "    state = model.rnn_cell.zero_state(1, dtype=tf.float32)\n",
    "    \n",
    "    # Get the embedding of each word in the sequence\n",
    "    embedded_words = model.embeddings(X)\n",
    "\n",
    "    # Unstack the embeddings\n",
    "    unstacked_embeddings = tf.unstack(embedded_words, axis=1)\n",
    "\n",
    "    # Iterate through each timestep and append its cell state\n",
    "    cell_states = []\n",
    "    for input_step in unstacked_embeddings:\n",
    "        _, state = model.rnn_cell(input_step, state)\n",
    "        cell_states.append(state[0])\n",
    "        \n",
    "    # Stack cell_states to (batch_size, time_steps, cell_size)\n",
    "    cell_states = tf.stack(cell_states, axis=1)\n",
    "    return tf.tanh(cell_states)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feel free to modify here the input! \n",
    "dummy_review = \"When you first start watching this movie, you can tell its going to be a painful ride. the audio is poor...\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Processing new review\n",
    "indexed_review, seq_length = process_new_review(dummy_review)\n",
    "indexed_review = tf.reshape(tf.constant(indexed_review), (1,-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the cell states\n",
    "cell_states = VisualizeRNN(lstm_model, indexed_review)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0kAAAFWCAYAAAC8ZtYnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmcZFV58PHf0z0LAwyI7IsLGJAoiguLcUURxcSEaEQwJk4UM1FjMDEmakzEJcmrMZoQzauOLAF8oyKJcVQUEAXjCoPKKiriwgiyzQgDM8PMdD/vH/c2c6ftmS66655b3f37fj71map7q+o51VPLfe455zmRmUiSJEmSKkNdN0CSJEmSBolJkiRJkiQ1mCRJkiRJUoNJkiRJkiQ1mCRJkiRJUoNJkiRJkiQ1mCRJkiRJUoNJkiRJkiQ1mCRJkiRJUsO8tgOsW78+247RtGGkaDgA1m4cLR6ztF1jXfmgIxvKxhteUDYeMLR2ddF4o4t2LhoPIIda/5rpXIxuKh5zaN1dZQN28BpjQ9nvnS4+H7/82AeKxnvQk59RNB7A6Jqy33MAw7vsUTReLNyuaDyA3LixaLybz/1E0XgAQwvK/n5suPveovEADviX/4ziQadpweNfMaWD7Q3fOWNGvVZ7kiRJkiSpYfaf4pUkSZLUFzE03HUTijBJkiRJktQTkyRJkiRJajBJkiRJkqQGkyRJkiRJaohhkyRJkiRJut+QPUmSJEmStJnD7SRJkiSpwSRJkiRJkhpiaKjrJhRhkiRJkiSpJ/Yk1SLiYOA4YF8ggZuB5Zn5vZbbJkmSJEnFbbO/LCLeCHwcCOAy4PL6+sci4k3tN0+SJEnSoIih4SldZprJepJOAh6dmRubGyPifcC1wLsmelBELAWWArz/Ax/gpJNO6kNTJUmSJHVpJiY8UzFZkjQK7AP8dNz2vet9E8rMZcAygHXr1+d0GihJkiRpMLiYbOXPgYsj4ofATfW2hwK/Bry2zYZJkiRJGiz2JAGZ+YWIOAg4gqpwQwArgcszc6RA+yRJkiQNCJOkWmaOAt8s0BZJkiRJA2zIJEmSJEmSNrMnSZIkSZIaTJIkSZIkqcEkSZIkSZIaTJIkSZIkqcEkSZIkSZIaXExWkiRJkhrmSk/SUNcNkCRJkjQzxNDwlC6TPm/EsRHx/Yi4ISLeNMH+hRHxiXr/tyLi4Y19j42Ib0TEtRFxdURsN93X2XpP0i/Xj7QdYgurC8cDuPu+TcVjlrZx8bTfaw/YyOjCovE2bMqi8QA2sEfReLffsaFoPIDFC6JovPnDZeMBzB8q//lYMLyoaLzttyt/Tm2Hncv+X24cLRoOgJ0e89ii8VYffEzReNDN33XdprJBS8cDuHdD2eOdw05+RNF4ABt3eUjReJ/5waqi8QAOKB5x+troSYqIYeDfgWOAlcDlEbE8M69r3O0kYHVm/lpEnAi8GzghIuYBHwX+MDOvjIhdgY3TbZM9SZIkSZJ6MjQUU7pM4gjghsy8MTM3AB8Hjht3n+OAs+rr5wFHR0QAzwGuyswrATLzzsyc9lkEkyRJkiRJXdoXuKlxe2W9bcL7ZOYm4C5gV+AgICPigoj4dkT8dT8aZOEGSZIkST2JyXuFJn5cxFJgaWPTssxcNrZ7goeMnwextfvMA54KHA6sBS6OiCsy8+IpNbRmkiRJkiSpJ9UItweuToiWbWX3SqA5CW0/4Oat3GdlPQ9pZ2BVvf3SzLyjbt/5wBOAaSVJDreTJEmS1JOW5iRdDhwYEftHxALgRGD5uPssB5bU118EfCkzE7gAeGxEbF8nT88ArmOa7EmSJEmS1JOpDrfblszcFBGvpUp4hoEzMvPaiHgHsCIzlwOnA+dExA1UPUgn1o9dHRHvo0q0Ejg/Mz833TaZJEmSJEnqSRtJEkBmng+cP27bWxvX1wPHb+WxH6UqA943JkmSJEmSejI0xTlJM41JkiRJkqSetNWTNGhMkiRJkiT1xCRJkiRJkhp6qFQ3K5gkSZIkSepJzJEFhKb8MiPi5f1siCRJkqTBFhFTusw008kF3761HRGxNCJWRMSKj/7HGdMIIUmSJGlQtLSY7MDZ5nC7iLhqa7uAPbf2uMxcBiwDuOWX9+aUWydJkiRpYFi4obIn8Fxg9bjtAXy9lRZJkiRJGkgmSZXPAjtm5nfH74iIS1ppkSRJkqSB5GKyQGaetI19v9//5kiSJEkaVHOlJ2mOFPGTJEmSpN64TpIkSZKknsyVniSTJEmSJEk9mYnlvKfCJEmSJElST2biwrBTYZIkSZIkqScxRyoamCRJkiRJ6onD7SRJkiSpwcINkiRJktTgnKQ+Wbyg7MDFhcPl/+P22bFsrrlhJIvGA1jQwd91h3tvLRswR8vGA5g3v2i4A/fZtWg8oPjg5Y3lPx6MjJYPuoiNZQPmSNl4wNC9q8sG7OCH/5dXX1U03oMWLioaD2Bk9W3FYzJa9v0a2+1QNB5AFP79uPoDHysaD+CXP72raLzrr7+zaDwA1t1QPuY0OdxOkiRJkhocbidJkiRJDcMmSZIkSZK0mUmSJEmSJDWYJEmSJElSg0mSJEmSJDWYJEmSJElSwzyTJEmSJEnabK70JJVd5VGSJEmSBpw9SZIkSZJ6Mld6kkySJEmSJPVkeGhuDESb9FVGxMERcXRE7Dhu+7HtNUuSJEnSoBkeiildZpptJkkRcTLwaeDPgGsi4rjG7n9ss2GSJEmSBotJUuWPgSdm5u8CRwF/FxGvq/dt9dVGxNKIWBERK8444/T+tFSSJElSp+ZKkjTZnKThzLwHIDN/EhFHAedFxMPYRpKUmcuAZQD3rF2XfWqrJEmSpA4Nx8xLeKZisp6kX0TE48Zu1AnT84HdgMe02TBJkiRJg8WepMrLgE3NDZm5CXhZRHy4tVZJkiRJGjgzMeGZim0mSZm5chv7vtb/5kiSJEkaVPNMkiRJkiRps7nSkzQ3VoOSJEmSNG1tzUmKiGMj4vsRcUNEvGmC/Qsj4hP1/m9FxMPr7UdExHfry5UR8YJ+vE57kiRJkiT1pI2epIgYBv4dOAZYCVweEcsz87rG3U4CVmfmr0XEicC7gROAa4DDMnNTROwNXBkRn6nrKEyZPUmSJEmSetJST9IRwA2ZeWNmbgA+Dhw37j7HAWfV188Djo6IyMy1jYRoO6Avyw+ZJEmSJEnqyVSTpIhYGhErGpeljafdF7ipcXtlvY2J7lMnRXcBuwJExJERcS1wNfCq6fYigcPtJEmSJPVoqsPtMnMZsGwruyd60vE9Qlu9T2Z+C3h0RPw6cFZEfD4z10+poTV7kiRJkiT1pKXhdiuBhzRu7wfcvLX7RMQ8YGdgVfMOmfk94F7gkGm8RMAkSZIkSVK3LgcOjIj9I2IBcCKwfNx9lgNL6usvAr6UmVk/Zh5ARDwMeCTwk+k2qPXhdveN9GXuVM8WDs/+2u2jZf+kAGwcLR9zdPtdisbL4flF4wEMbVxXNF6MbCwaD2Bk3nZF420cKf9mveu+keIx78iy57h2XFD+nFou3KNovC4WSHzwMc8vG3DxbmXjAcO77lU85ui9dxeNF0PDReMBDC1+UNF4+z3j0UXjAWz/vZ8Wjbf+ujuKxpup2qhuV1emey1wATAMnJGZ10bEO4AVmbkcOB04JyJuoOpBOrF++FOBN0XERmAUeE1mTvs/0zlJkiRJknrS1mKymXk+cP64bW9tXF8PHD/B484Bzul3e0ySJEmSJPWkrSRp0JgkSZIkSerJcJgkSZIkSdL9hkySJEmSJGmzOVAjDTBJkiRJktSjIeckSZIkSdJmzkmSJEmSpAbnJEmSJElSg3OSJEmSJKnBOUmSJEmS1OBwu1pEHAFkZl4eEY8CjgWuz8zzW2+dJEmSpIHhcDsgIk4BngfMi4iLgCOBS4A3RcTjM/Mf2m+iJEmSpEFgT1LlRcDjgIXAL4D9MvPuiHgP8C3AJEmSJEmaI4bnyJykoUn2b8rMkcxcC/woM+8GyMx1wOjWHhQRSyNiRUSsOOvMM/rYXEmSJEldGYqY0mWmmawnaUNEbF8nSU8c2xgRO7ONJCkzlwHLAO5cszb70VBJkiRJ3XJOUuXpmXkfQGY2k6L5wJLWWiVJkiRJHdlmkjSWIE2w/Q7gjlZaJEmSJGkgzcShc1PhOkmSJEmSejJXCjeYJEmSJEnqyRzJkUySJEmSJPVm2OF2kiRJkrSZc5IkSZIkqWF4slVWZwmTJEmSJEk9sSdJkiRJkhqckyRJkiRJDfYkSZIkSVKDc5L6ZP2mbDvEFjaMlI0HsGhe2Yy6g5fISJYP+pP1ZT+FmSNF41UWFI3WRRf5xtENReN18FZl0fzyf9fSr3PdxtGyAYG1G8u+yAXD5f8fX7ticdF4L37i7kXjAdx2787FY5Y+0z3awQ/zwrvL/kZ+8O7nFY0HsGpoTdF4t//GgUXjAfxj8YjTZ0+SJEmSJDXMkRzJJEmSJElSb4aYG1mSSZIkSZKkntiTJEmSJEkNQyZJkiRJkrTZXOlJmiNF/CRJkiSpN/YkSZIkSeqJhRskSZIkqWGuDLczSZIkSZLUEws3SJIkSVLDHMmRHnjhhog4u42GSJIkSRpsQxFTusw02+xJiojl4zcBz4yIBwFk5u+01TBJkiRJg6WtfCcijgVOBYaB0zLzXeP2LwTOBp4I3AmckJk/qfe9GTgJGAFOzswLptueyYbb7QdcB5wGJFWSdBjw3ukGliRJkjSztLF+UEQMA/8OHAOsBC6PiOWZeV3jbicBqzPz1yLiRODdwAkR8SjgRODRwD7AFyPioMwcmU6bJnudhwFXAG8B7srMS4B1mXlpZl66tQdFxNKIWBERKz76H2dMp32SJEmSBkRETOkyiSOAGzLzxszcAHwcOG7cfY4DzqqvnwccHdUTHwd8PDPvy8wfAzfUzzct2+xJysxR4F8i4pP1v7dO9pj6ccuAZQA/X31vTreRkiRJkrrXUnW7fYGbGrdXAkdu7T6ZuSki7gJ2rbd/c9xj951ug3qqbpeZK4HjI+K3gLunG1SSJEnSzDPVOUkRsRRY2ti0rO5YgYmL5o3vaNnafXp57AP2gEqAZ+bngM9NN6gkSZKkmWeqc5KaI80msBJ4SOP2fsDNW7nPyoiYB+wMrOrxsQ9YG3OvJEmSJM1CLc1Juhw4MCL2j4gFVIUYxlfZXg4sqa+/CPhSZma9/cSIWBgR+wMHApdN93W6mKwkSZKknrQxJ6meY/Ra4AKqEuBnZOa1EfEOYEVmLgdOB86JiBuoepBOrB97bUScS1WRexPwp9OtbAcmSZIkSZJ61NaysJl5PnD+uG1vbVxfDxy/lcf+A/AP/WyPSZIkSZKknrRU3W7gmCRJkiRJ6kkP84tmBQs3SJIkSVKDPUmSJEmSeuJwO0mSJElqmCM5kkmSJEmSpN4MzZE5Sa0nSfdsnHaZ8gdku+Hy06xGRrNovOEO+jlvvXdj8Zg/u2t90XjzO/i7zi/8ft1lu/lF4wGMZtnPxw4LhovGA5jXwXsnKPt33TRaNBwAd6zdUDTeztuVP294w49WFY238sDdisYDuOqmu4rHXDS/7PfAhpHyH5Bdd1hQNN5NV32vaDyADfeWfe+UjjdTzZEcyZ4kSZIkSb2Jwic/u2KSJEmSJKk32cHQgg6YJEmSJEnqSZgkSZIkSVKDSZIkSZIkNTgnSZIkSZIa7EmSJEmSpM2ckyRJkiRJTSZJkiRJktRgkiRJkiRJDSZJkiRJktQwapL0KyLiqcARwDWZeWE7TZIkSZI0iOZK4Yahbe2MiMsa1/8Y+ACwGDglIt7UctskSZIkqbhtJknA/Mb1pcAxmfl24DnAS7f2oIhYGhErImLFJ84+sw/NlCRJktS5HJ3aZYaZbLjdUETsQpVMRWbeDpCZ90bEpq09KDOXAcsAvn/b3XNjWV5JkiRptsu5cWg/WZK0M3AFEEBGxF6Z+YuI2LHeJkmSJGmumIG9QlOxzSQpMx++lV2jwAv63hpJkiRJA2uuFG6YUgnwzFwL/LjPbZEkSZI0yEySJEmSJKnBJEmSJEmSGkySJEmSJGkz5yRJkiRJUtOoSZIkSZIkbeY6SZIkSZLU4HA7SZIkSdrMOUmSJEmS1GSS1B+7LSqbh80biqLxADoIWdwuC+YXj/mYDTcWjRejm4rGA2BkY9l49w2XjdeBHN2ueMyRebsVjzl8zx1F48XIhqLxAPZfuEPReEP3rSsaD+Abv3lv0Xj3/fDsovEAfvmDm4rHHNlY9vv8npVlP48AIxvLHqg+58ZVReMBfO6a24vGu2fT3Dj4nzaTJEmSJElqGB3pugVFmCRJkiRJ6knOkRLgQ103QJIkSZIGiT1JkiRJknrjcDtJkiRJapgjSZLD7SRJkiT1JEdGpnSZjoh4cERcFBE/rP/dZSv3W1Lf54cRsaSx/QsRcWVEXBsRH4qIScv9miRJkiRJ6s3o6NQu0/Mm4OLMPBC4uL69hYh4MHAKcCRwBHBKI5l6cWYeChwC7A4cP1lAkyRJkiRJvRkdmdpleo4DzqqvnwX87gT3eS5wUWauyszVwEXAsQCZeXd9n3nAAiAnC2iSJEmSJKknOToypcs07ZmZtwDU/+4xwX32BZqrV6+stwEQERcAtwFrgPMmC2jhBkmSJEm9meLQuYhYCixtbFqWmcsa+78I7DXBQ9/Sa4gJtt3fY5SZz42I7YD/BzyLqqdpq0ySJEmSJPVkqr1CdUK0bBv7n721fRFxa0TsnZm3RMTeVD1C460Ejmrc3g+4ZFyM9RGxnGr43jaTpG0Ot4uIIyNip/r6ooh4e0R8JiLeHRE7b+uxkiRJkmaZbuYkLQfGqtUtAT49wX0uAJ4TEbvUBRueA1wQETvWiRURMQ/4TeD6yQJONifpDGBtff1UYGfg3fW2Myd7ckmSJEmzSDfV7d4FHBMRPwSOqW8TEYdFxGkAmbkKeCdweX15R71tB2B5RFwFXEnVC/WhyQJONtxuKDM31dcPy8wn1Ne/GhHf3dqDmmMO33vq+1ny8ldM1g5JkiRJA266ax5NKWbmncDRE2xfAbyycfsMqk6e5n1uBQ5/oDEnS5KuiYiXZ+aZwJURcVhmroiIg4CNW3tQc8zhnWvWTlpiT5IkSdIMMP2hczPCZEnSK4FTI+JvgTuAb0TETVTl9V65zUdKkiRJml1MkiAz7wL+KCIWAwfU919Zd1tJkiRJmkNy+vOLZoSeSoBn5hqqiU6SJEmS5ip7kiRJkiSpwSRJkiRJkjabK8PtJlsnSZIkSZLmFHuSJEmSJPXG4XaSJEmS1GCSJEmSJEmb5YhJkiRJkiRtNkcKN5gkSZIkSeqNw+36YzTbjrCl+zaVz263n1+2SOCm0n9UYF0Hn4fh3Q4uGm9HNhSNBxD33VM24FAHBS2HCp+L2dTB/2OW/94Z2WmPovFy4eKi8QAyyr5fY2Rj0XgAl2x8aNF4R77weUXjAeyR5T+TDA2XjdfBe2f4ntuLxvvFdvsWjQfw18NRNN72n31v0XgzVZokSZIkSdJmc2WdJJMkSZIkST3JEZMkSZIkSbqfSZIkSZIkNTjcTpIkSZIa7EmSJEmSpAaTJEmSJElqGB2xBLgkSZIk3c85SZIkSZLU4HA7SZIkSWqYK0nSUNcNkCRJkqRBss0kKSJOjoiHlGqMJEmSpMGVo6NTusw0k/UkvRP4VkT8b0S8JiJ2L9EoSZIkSYNndGR0SpeZZrIk6UZgP6pk6YnAdRHxhYhYEhGLt/agiFgaESsiYsXZZ57Rx+ZKkiRJ6kqOjE7pMtNMVrghM3MUuBC4MCLmA88DXgL8MzBhz1JmLgOWAdx+99rsX3MlSZIkdWUmJjxTMVmSFM0bmbkRWA4sj4hFrbVKkiRJ0sCZifOLpmKyJOmEre3IzHV9boskSZKkAWZPEpCZPyjVEEmSJEmDzSRJkiRJkhpGHW4nSZIkSZvZkyRJkiRJDTky0nUTijBJkiRJktQTq9tJkiRJUoPD7SRJkiSpwSRJkiRJkhpGTZIkSZIkaTPnJPXJ9XesazvEFrabN1Q0Xhfuum9T8Zi33XNf8Zg/XrW2aLxFC4aLxgO4Z33Z/8sH77igaDyAnReW/TKdPxxF4wHMHyp/vunGwp+PdRvWFI3XhdvuLv89d+m3bioa7yEH7FI0HsCaNeX/rln4GG7+wvK/H5s2lq0wtuK8fysaD2BkQ9ljyL0OfWbReAA/e2nxkNPWxXC7iHgw8Ang4cBPgBdn5uoJ7rcE+Nv65t9n5ln19gXAB4CjgFHgLZn5X9uKOfszCkmSJEkz2ZuAizPzQODi+vYW6kTqFOBI4AjglIgYOzP0FuC2zDwIeBRw6WQBHW4nSZIkqSc5kl2EPY6qFwjgLOAS4I3j7vNc4KLMXAUQERcBxwIfA14BHAyQmaPAHZMFtCdJkiRJUk9GR0andImIpRGxonFZ+gDC7pmZtwDU/+4xwX32BZpjmFcC+0bEg+rb74yIb0fEJyNiz8kC2pMkSZIkqSc5OrWepMxcBizb2v6I+CKw1wS73tJjiIkmJSdVvrMf8LXMfH1EvB74Z+APt/VkJkmSJEmSejLa0nC7zHz21vZFxK0RsXdm3hIRewO3TXC3lWwekgdVYnQJcCewFvhUvf2TwEmTtcfhdpIkSZJ6kiOjU7pM03JgSX19CfDpCe5zAfCciNilLtjwHOCCzEzgM2xOoI4GrpssoD1JkiRJknrSUeGGdwHnRsRJwM+A4wEi4jDgVZn5ysxcFRHvBC6vH/OOsSIOVEUezomIfwVuB14+WUCTJEmSJEk9aWu43bZk5p1UPUDjt68AXtm4fQZwxgT3+ynw9AcS0yRJkiRJUk+6WEy2CyZJkiRJknoyOsXqdjONSZIkSZKknnQ0J6m4bSZJEbEAOBG4OTO/GBG/DzwZ+B6wLDM3FmijJEmSpAEw6nA7AM6s77N9RCwBdgT+m2ri1BFsLsUnSZIkaZazJ6nymMx8bETMA34O7JOZIxHxUeDKrT0oIpYCSwHe8Pfv5XdeYi4lSZIkzXQmSZWhesjdDsD2wM7AKmAhMH9rD8rMZcAygP+98c658ZeUJEmSZjmH21VOB64HhoG3AJ+MiBuBJwEfb7ltkiRJkgZIWt0OMvNfIuIT9fWbI+Js4NnARzLzshINlCRJkjQYulhMtguTlgDPzJsb138JnNdqiyRJkiSpQ66TJEmSJKkn6ZwkSZIkSdrM6naSJEmS1OCcJEmSJElqyFGH20mSJEnS/exJkiRJkqQG5yRJkiRJUoPV7SRJkiSpweF2fbT7DvNLhAFgzX0j7LN4QbF4ADev2cB+O5WLedftmzho10XF4gHcds99PGqPHYvG/PGqtRy234OKxbv2tjU8bu+disUD+OqPV/GMA3YtFu/q29bw9Ic9uFg8gCt/sYaDd9uhWLwfrV7LHjuU/Q5YvW4je+ywsGjMG1et5bF7lXu/futnq3nmI3YrFg/gyz+6g2cfWC7mf17xc1546N7F4gFc+q2bGJ4XRWO+6LD9isY788s/4g+efkDRmOdcciN/csyBxeKd8ZUb+cvnPrJYPIB3f/Y6/qxgzCXnwW4HHV4sHsCt13yFfQ//zWLxRjasY+8D9ioWb6aaK8PtIrPdF3r9rXcX/UsuXjBcMhwAQ2V/39g4Wv7NuWrdpuIxb71nQ9F484cL/0cCw1E25i6Lyp2wGHPfprLd8ms2lH+vzh8aKh5z7caRovG2n1/+u3W48J91feH3KsDrz7i8aLyTX3hI0XgAGzs4oCr9fu3iu7X0d8BfvufCovEA5i0qe3J2j/12LhoP4FtvPab8wcc0Ld/7kCl9qH/nlmtm1Gt1uJ0kSZKknoy03MEyKEySJEmSJPVkjoy2M0mSJEmS1Bt7kiRJkiSpwZ4kSZIkSWqwJ0mSJEmSGuZKT1L5urWSJEmSNMDsSZIkSZLUE4fbSZIkSVLDXBluZ5IkSZIkqScmSbWIeATwAuAhwCbgh8DHMvOultsmSZIkaYDMleF22yzcEBEnAx8CtgMOBxZRJUvfiIijWm+dJEmSpIExklO7zDST9ST9MfC4zByJiPcB52fmURHxYeDTwONbb6EkSZKkgWBP0mZjidRCYDFAZv4MmL+1B0TE0ohYERErzj3nzOm3UpIkSVLn7EmqnAZcHhHfBJ4OvBsgInYHVm3tQZm5DFgGcP2td8/AP4skSZKk8eZKT9I2k6TMPDUivgj8OvC+zLy+3n47VdIkSZIkaY6Yib1CUzFpdbvMvBa4tkBbJEmSJA0we5IkSZIkqWG06wYU0kvhBkmSJEliJHNKl+mIiAdHxEUR8cP63122cr8l9X1+GBFLGttPiIirIuLaiPinXmKaJEmSJEnqSUfV7d4EXJyZBwIX17e3EBEPBk4BjgSOAE6JiF0iYlfgPcDRmfloYM+IOHqygCZJkiRJknrSRU8ScBxwVn39LOB3J7jPc4GLMnNVZq4GLgKOBQ4AflAXngP4IvB7kwV0TpIkSZKknnRU3W7PzLwFIDNviYg9JrjPvsBNjdsr621fAA6OiIfX234XWDBZQJMkSZIkST2Zaq9QRCwFljY2LavXVh3b/0Vgrwke+pZeQ0ywLTNzdUS8GvgEVd2Jr1P1Lm2TSZIkSZKknky1J6lOiJZtY/+zt7YvIm6NiL3rXqS9gdsmuNtK4KjG7f2AS+rn/gzwmfq5lgIjk7XXOUmSJEmSBtlyYKxa3RLg0xPc5wLgOXWxhl2A59TbGBueV29/DXDaZAHtSZIkSZLUk44Wk30XcG5EnAT8DDgeICIOA16Vma/MzFUR8U7g8vox78jMVfX1UyPi0Mb2H0waMTMH8gIsNebMj+drnB3x5kpMX+PsiDkXXqN/19kRz9c4e2J28Rq9tHsZ5OF2Sye/izFnQLwuYvoajTlT4nUR09dozJkSr4uYvkZjzpR4atkgJ0mSJEmSVJxJkiRJkiQ1DHKStNUSgcacUfG6iOlrNOZMiddFTF+jMWdKvC5i+hqNOVPiqWVRTzaTJEmSJDHYPUmSJEmSVJxJkiRJkiQ1mCRJkiRJUoNJUmER8bpetvU55sJetmlmiYi9/X/UoIqIPSPi+fVlj67b05aI2CUijoiIp49dum4UVV45AAAW/UlEQVSTZoaIWBARh9SX+V23p58iYigiXtx1O6TpGKjCDRHxFOBtwMOAeUAAmZkHtBTvZRNtz8yz24hXx/x2Zj5h3LbvZObjC8f8lW19jPcZYPwb6y5gBfDhzFzfQszXAWcCa4DTgMcDb8rMC/sdqxHzIOCDwJ6ZeUhEPBb4ncz8+7Zijov/ReARwH9l5htaeP7in4867jCwJ9V3wFjMn/U5xtX86nsUNn/nPLaf8SaI/whgZWbeFxFHAY8Fzs7MX7YU78ETbF6TmRtbivdi4D3AJVR/06cBf5WZ57URr465J/CPwD6Z+byIeBTwG5l5eosxXwm8DtgP+C7wJOAbmfmsPsd5PxO/XwHIzJP7GW9c7E6/5xrt2Cszf9HSc3fx3jkKOAv4CdVn5CHAksz8SkvxngJ8NzPvjYg/AJ4AnJqZP20jXh3zK5npSQPNWIOWJF0P/AVwBTAytj0z72wp3vsbN7cDjga+nZkvaiHWS4DfB54K/G9j107Apsx8dgsx9wL2BT5ax45GzA9l5sH9jlnHPRXYHfhYvekE4BfAImCnzPzDFmJemZmHRsRzgT8F/g44s61EsI55KfBXVInf4+tt12TmIW3FnKANATwqM69t4bmLfT4aMf8MOAW4FRitN/c9aYmIh21rf5sHDnX87wKHAQ8HLgCWA4/MzN9sKd5PqA7CVlN9DzwIuAW4DfjjzLyiz/GuBI7JzNvq27sDX8zMQ/sZZ1zMz1OdKHlL/V0wD/hOZj6mxZhXA4cD38zMx0XEwcDbM/OEPsdZsq39mXlWP+ONi93591wd83OZ+VstPXcX750rgN/PzO/Xtw8CPpaZT2wp3lXAoVQnZM4BTgdemJnPaCNeHfPvgHXAJ4B7x7Zn5qq2Ym6lHW/LzLe1+PzbAzcDv5eZF7cVR+XNm/wuRd2VmZ8vFSwz/6x5OyJ2pvryaMPXqQ5KdgPe29i+BriqpZjPBf6I6izne9mcJN0N/E1LMQEeP+7s0WfGzihFRN8P5mtjr+03qZKjK+sEok3bZ+Zl48JsajnmFrI6y9HK37Tw52PM66iShVZOjIxpOwnqwWhmboqIFwD/mpnvj4jvtBjvC8CnMvMCgIh4DnAscC7wf4Ej+xxvaCxBqt1J+8O7d8vMcyPizQD133dksgdN0/rMXB8RRMTCzLw+Ih7Z7yBtJkE96Px7DqCtBKnWxXtn/liCVMf8QctD7jZlZkbEcVQ9SKdPlnz3wSvqf/+0sS2BVkYHbUNfTwJN4MVUv8OvBEySZpFBS5K+HBHvAf4buG9sY2Z+u1D8tcCBbTxxfVD204h4NrAuM0frM0cHA1e3FPOsiDgHeElm/r82YmzF7hHx0LEhUhHxUKrkEGBDSzGviIgLgf2BN0fEYjb3RLTljnrYVAJExIuoEuHZqrXPR8NNVEMzWxURa9g8fGns6C/ZPNxup5absLHuXV4C/Ha9rc0DpMMy81VjNzLzwoj4x8x8fUvz2j4fERewZW/y+S3Eabo3InZl8+fxSbT/XloZEQ8C/ge4KCJWU51RbkVEfJkJht31e3jfOHPhe66L986KiDidzSeeXkq7B/Nr6iTwD4Cn18OaW50HlZn7t/n8vcrMz7Qc4hVUCdJ/R8Qumbm65XgqZNCSpLGzmYc1tiXQyg/AuLkzw8CvU51ZbdNXgKdFxC5UZxxWUB1AvLSNYHUy9idAySTpL4GvRsSPqA469wdeExE7UI3BbsNJwOOAGzNzbf2D9/KWYo35U6oVtg+OiJ8DP6b6AZoVOvp83AhcEhGfY8sTJe/rZ5DMXNzP55uClwOvAv4hM38cEftTDYtty6qIeCPw8fr2CcDq+kCpjZMJCXyYanhxUH1OntRCnKbXUw1bPCAivkY15Le1oaEAmfmC+urb6gRmZ6peu7Y05x5uB/we7ffqzOrvudrYe+cRpd47wKup/rYnU31GvkLVq9uWE6iG3Z+Umb+oT16+p8V41D1jrwbGRpZcQjVss5W5kNtox/Mz87MtPffBVD3n34uIj1F9Nt4/ycM0QwzUnKTSIqI5FncT8NPMXNlyzG9n5hPquReLMvOfov3CDcXHBddnpw+m+vK/vo1iDXWcg+shLhPOPSrRC1knf0OZuabtWCV19Pk4ZaLtmfn2FmM+FTgwM8+MiN2AxZn547bi1TFfl5mnTratj/F2o5rrNZa0fBV4O9XZ8odm5g19jjdRsZir2iyIERHbAa+lGma8BvgG8P62vnsGRURc2ua8kkacWfk9N6aeh/RIqs/H90sfyM9GEXEaVW/V2MnRPwRGMvOVhdvx9syc8LelD8/9HqpjnNPrxPN/2pwLrbIGKkmKbirM7Ek18RbgsnHj6NuI9x3gNcC/UJ3RuTYirm55guhEB3yZLVUNrGM+mWpSerNCWd+rokXEssxcWp/FHS/bGIYSEX+QmR+NiNdPtL/fvR5dKv35KK1Oyg6jmgd1UETsA3wyM5/SctziVS5LiIhXU32/HQD8qLFrMfC1zGytByIizqWabznWa/4SYJfMPL6tmKXFllUKh4AnAv+WmX2fB7W177cxs+l7Dor+Zp2bmS+OrVTY7PeJhIj4amY+ddwQYygwtDjqgkqTbWtbtFQZse4p+z7wmMy8t952EfDmzFzR73gqb9CG2/0HdYWZ+vYPqHo/WkmS4lfL1L4/IlotU0s1Mf3NVJOor42IA4CJDvD7pvS44Hoe1COoSuKOTX5NoO8/OJm5tP73mf1+7m3Yof636yFbreri8xFVFbS/Bh5NNZwIaHXOxQuoysV/u45zcz2frRWxucrl/hGxvLFrMVVxg7biHkQ1VOvhbHkQ2O+/638Cnwf+D/CmxvY1BSpaPXLcwdeXo6qyN5tcwea5c5uohr6d1FKssc/BI6lOlIy9X3+bamjYrFHyN4vqGADg+S0896/IzKfW/3bxezUSEY/IzB8B1Mc7bRfEmMhptPP3nk9VIfDexrZX0s1rVAsGrSfp8sw8vHlGNSK+m5mPayle0TK19fj/d2XmX7Xx/JPEPgR4FFseeLay3k1EfI+qLHXRN1epM4GNeLtn5u1tPX/XSn8+6hgXUp0YeQPVnJ0lwO2Z+caW4l2WmUc0hsHuQLXOTSvDwqIqPb4/EyQRwFWZ2cr8kvr/8kP86vIKbVd9KiYi/oNqaYNv1rePpFp35jWdNqwPIuL4zPxkRByQmTcWjn0hVWnjNfXtxVS9rceWbEebuvrNmu0i4miqE983UiX2DwNenpmtnhgurf5tZDYfD8xVg9aTVLrCTNEytZk5EhGtrIGwLfWQoqOokqTzgedRzUloK4G4BtiLghWQCp8JHPP1eijjJ4D/noUVbboo47xrPbb7dZl5KXBpVOu0tOXciPgw8KCI+GOqKkUfaStY1lUugd9oK8ZWbMrMDxaOWURj2NJ84GUR8bP69sOA67psWx+9GfgkcB7VIqAlPZQtq5JuoDoZNZsU+82aYNjbFtoc/lZaZl4cEQeyea7X9Zl53yQPmxEiIqjmeb6W6rUNRcQmqnmQ7+i0ceqbQUuSSleY6aJM7XfqYTafZMsiCv/dYswXUS0i953MfHk9z+S0FuPtBlwXEZexZYWy32kx5mEUPhOYmQdGxBHAicBbIuI64OOZ2WaVspK6+HyMTZa+JSJ+i6qc8n4txtud6sDzbqof8rcCfV/YeUyH8wM+ExGvAT7Flp/Joos6tqTIsKWO3VnPuxw/TBNo/bv1HOCyiPgU1Xv2BbR78qmY2FzBczGFfrPGhr1FxDuoFlk/h+rz/1Jm2RDues7On9CobhcRxavbteTPgacAh48V+qmHE34wIv4iM/+l09apLwZquB2UrTATESdTrcvytDreVzLzU23Fq2OeOcHmzMxXTLC9XzHHhhRdATyTamjPNZn56JbiTVhpqe4ZaEVEfBI4OTM7Wb8jquph7wNempnDXbSh3zr6fDwf+F/gIVRlVHcC3pYtrXOxlQIKrVZh60IXxVvUPxGxgKoH6RyqOQ9baPO7tY7/BKrvAai+B9pc+LiY+rcqgHdTzYW8fxfw7szs9yLLzdjfGv/8E22byWJAqtu1IaoiXMdk5h3jtu8OXDjTi/CoMmg9SQBHsHleyRMios15JXtQrVHwbeAM4IKW4twvM9teu2ciK6Ja8PAjVHMS7gEuaytY2z/YTV2cCWzE3onqrOqJVEP9PkX1/p0tin8+gOOBr2bmNcAz62pe/wz0NUmKRhW2iLiqsWsx8LV+xhoEpYu3qL8ycwPwzYh4cql5DxGxU2beXX8Gf1JfxvY9eDb0Qo79VkXE/PG/WxGxqOXwIxHxUqq1y5KqGuNsm/B/+Lg5rF+aRcVU5o9PkKCal1T3oGkWGKiepK3NK8nMk1uMGcBzqBZ3PIxqsczTx6qx9DHOX2e1JtL7mbjsZ2uvcVw7Hg7slJlXTXLXqTx38aFEHZ8J/DHwP8C5mfmNtuJ0qdTnoxHvV8pgT7StD3F2BnahmypsxUTEszLzSxHxwon2tzzMV30W5aoUEhGfpapkN0IjQWLz9/mM74WMbkvWPxw4lWrIVlKdnPnzzPxJWzFLi4hvA8ePq2533vje+5loolEIvezTzDJoPUldzCvJiPgF1djgTVQHTudFxEWZ+dfbfvQD8kbgn6i+iItO8I+IizPzaICxL+Dmtn7JDkqNdnwm8ID6/bM4InbMzHtajldcwc/HmKGI2GWsCEZ9Frvv31OZeRdVUZiX9Pu5B8wzgC9RHeyOl4BJ0szySaoqhafRcq9DZj4f7q8wO1sP+DorWV//Fh/XZowB8FdUpfjHKjI+nOqE22xwaETcPcH2oFFFWDPboCVJRaui1XMulgB3UP3o/FVmboyIIeCHbNkzMV23RlX+9+VU84JaF9UK9NsDu0XELlQfXqjmeezTcuxhYE+2PNv5sxbidDls6tF17+eDq6bE7VQlh69pOW4RhT8fY95LVTXwPKqD+BcD/9BCnDkh61XmOxrmq/7rokrh1yPi8My8vHDc1nV5sqSenzzRqJLW5id34GvAh4GxE7IfBmbFqIvZMvdY2zYQSVKH80p2o1oI7KfNjZk5Wk8g76cPAl+g6tZvrsQcVK+9jaELf0JVgWUfqrlIY7HWAB9oIR4AEfFnVKUxbwVG680JtDEZvsvFK5cBr896zYeIOKre9uSW45ZS8vMx9txnR8QK4FlU79cXZuZsKePcmXp44SlsrjJ1KfCO+iBRM0cXVQqfBbwqIn5KVZF1bLjdrCpu0oHPNq5vRzW/9eaO2tKWs6kqh76zvv0SquIjx3fWIukBGIg5SRHxF1QJ23fYXAL4fiULAbQtIj6Yma8uHPOtwL/Wk3D/jqpK0jsz89stxbsBODIz72zj+QdFRFw5blLqhNukrkXEf1H11DerTB2amRPOVdJg6qJKYT0CYqKgP51ou6am7qH/Yhvzy7oym38jG3Ovo7E5qY5lF2TmQHRCaHoG5T9xX6qz738DXAl8naqb9huzaRI1QOkEqfaizHxHRDwVOIZqSNMHgbaKGtxEu4sAD4ob66TznPr2HwATHcRIXXtEZv5e4/bbI+K7nbVGU9JFlUKToWIOpFq4dzb5TkQ8KTO/CRARRzJLqoeOn3sdEYuphv7/CVVPr2aBgUiSMvMNcP9aEIdRJUyvAD4SEb/MzEd12b5ZYGyC728BH8rMT0fE2/odJCJeX1+9kWrRuM+x5ZCQ9/U7ZsdeAbwd+C/qdYSAP+qyQdJWrIuIp2bmVwEi4inAuo7bpB5ZpXD2GVcFNqmGp7cxz7NLRwIvi4ix+cgPBb4XEVczS4Zs1sur/DnwMqrh/4fP9lE0c8lAJEkNi6iKCuxcX24Gru60RbPDzyPiw8CzgXdHxEJgqIU4Y2dWflZfFtSX2eoRVIueDlF9lo6mGr8/47/4Neu8Cji7npsEVYXNJR22Rw+MVQpnmcxcXFfvPJDN1dC6n//QX8d23YC2RLWA/F8CJ1CtI/h453jOPoMyJ2kZ8GiqggLfAr4JfHOsDLCmJyK2p/qyujozfxgRewOPycwLO27ajBYR36das+QaNheocHiKBk6jl3fH+t97qIbEXpGZDruTCouIVwKvA/ajWhvySVRTDGbNnKTZLCLuBW4HzqQ6dt3CLBw5MycNSk/SQ4GFVGWFfw6sBH7ZaYtmkcxcS+NMY2beQotl1iPiIqoF5H5Z394F+HhmPretmB25PTM/03UjpB4cVl+WUw0N/X3gcqqqZZ/MzH/qsnHqXUT8FtVJxfvXYsnMd3TXIk3R64DDqU4IPzMiDqYavq2Z4T1s7vkrtjakyhqIJCkzj42IoPrifzJVF+YhEbGK6szKKZ02UA/U7mMJEkBmro6IPbpsUEtOiYjTgIvZcu6VQ180aHYFnjC24HFEnAKcR1US/Aqqha414CLiQ1Rr3z2Tau2yFwGXddooTdX6zFwfEUTEwsy8PiIe2XWj1JvMfFvXbVD72piXMiVZuQY4n2rdm69Rzfl4XacN01SMRMT9VXrqErLdj+vsv5cDj6Mayvjb9aWV9YOkaXoosKFxeyPwsMxcRyPB18B7cma+DFidmW8HfoNqXqRmnpX1pP//AS6KiE8z+9ZJmrUi4tzG9XeP2+dUhlliIHqSIuJkqh6kp1D9eH+NalXmM7Bww0z0FuCrETG2vtXTqcpizjaHZuZjum6E1IP/BL5ZH4hBldB/LCJ2AFysd+YYq0i4NiL2Ae4EipcF1/Rl5gvqq2+LiC9TFav6QodN0gNzYOP6McAbG7d3L9wWtWQgkiTg4VRDP/6ini+jGSwzvxART6CaiBpU/693dNysNnwzIh6VmR5kaqBl5jsj4nzgqVSfyVdl5op690u7a5keoM/WvQ//RDVMEqphd5rBMvPSye+lAbOt0TGzceTMnDQQ1e00u0TExZl59GTbZrqI+B7VkNAfUw1ZCmbJ2g+SBk9ELAJeDTyN6kDsf4EPZub6ThsmzTERcT3wEqppKx+lKoYT9eWjmfnrHTZPfWKSpL6JiO2oJhV/GTiK6ssCqrWvPj/bvjTquVa/whLgktpQz4NYQ3VQBtVB2oMy88XdtUqae+ohkluVmc8s1Ra1xyRJfRMRr6NaeXofqlLuY0nS3cBHMvMDXbVNkma6iLgyMw+dbJskafoGprqdZr7MPDUz9wfekJkHZOb+9eVQEyRJmrbvRMSTxm5ExJFUhY4kFRYRiyJi/EmLh0bEvl21Sf1lT5JaERGHAI9iywUPz+6uRZI0s9XzIB8J/Kze9FDge8AozoeUioqI+cD1wGMz895624XA3zQK42gGG5TqdppF6oUqj6JKks4Hngd8FTBJkqSpO7brBkiqZObGiPgUcAJwRr0+5O4mSLOHPUnqu4i4GjgU+E5mHhoRewKnZeZvd9w0SZKkvoiIg6nmXD8tIv4WuDsz/63rdqk/7ElSG9Zn5mhEbIqInYDbgAO6bpQkSVK/ZOb1EUFEHERVbfKpXbdJ/WOSpDZcXi94+BGqBQ/vAS7rtkmSJEl9dzrVos5XZebqrhuj/nG4nfouIs4BvkK10OF6YKfMvKrbVkmSJPVXRGwP3AL8XmZ+sev2qH9MktR3EfEsqi7np1ENs/su8JXMPLXThkmSJEk9MElSKyJiGDgceCbwKmBdZh7cbaskSZKkyTknSX0XERcDOwDfoBpyd3hm3tZtqyRJkqTeDHXdAM1KVwEbgEOAxwKHRMSibpskSZIk9cbhdmpNROwIvBx4A7BXZi7suEmSJEnSpBxup76LiNdSFW14IvBT4AyqYXeSJEnSwDNJUhsWAe8DrsjMTV03RpIkSXogHG4nSZIkSQ0WbpAkSZKkBpMkSZIkSWowSZIkSZKkBpMkSZIkSWowSZIkSZKkhv8P6HsSn6CQyxYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fbe8acd2668>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot activations of the first 10 units in the cell (the cell has 64 units in total)\n",
    "plt.figure(figsize = (16,5))\n",
    "sns.heatmap(cell_states.numpy()[0,:,:10].T, \n",
    "            xticklabels=word_tokenize(dummy_review)+['<END>'],\n",
    "            cmap='RdBu');"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
